thu-coai/MTAC-IFBench

MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding

🌟 Overview

MTAC-IFBench benchmarks instruction following in multi-turn agentic coding.

Existing agentic coding benchmarks (e.g., SWE-bench, Terminal-Bench) focus on final functional correctness, while current instruction-following benchmarks confine themselves to single-turn chat or code generation. Neither answers the question that matters in a real development session: does the agent keep following the rules, turn after turn, as the requirements change?

In real multi-turn software development, an agent must comply with:

• —Repository policy files (CLAUDE.md, AGENTS.md) that govern the whole session
• —Per-turn constraints on the response, the code, the environment, and its own workflow
• —Constraints that persist across turns without decaying as context grows
• —Constraints that are added, revised, or overridden by later instructions

Each instance in MTAC-IFBench is a complete development session: a repository policy file, 5–10 progressive user instructions, a constraint checklist for every turn, and a function checklist for the finished project. Each checklist item is verified by a verification script or a judge agent otherwise. MTAC-IFBench identifies significant deficiencies in existing code agents in multi-turn instruction-following, with performance degrading rapidly as the interaction session grows longer. See our paper for full results.

📊 Data statistics

📦 Constraint taxonomy

Constraints in MTAC-IFBench span 6 primary and 18 secondary categories, covering the generated response and code as well as the agent's environment interactions and workflow.

⚙️ Data format

Each line is one instance, a JSON object with the following fields:

{
  "task_id": 1,
  "task_category": "Data Analysis",
  "repository_policy": "# 项目规范\n所有的回复内容都应以语气词\"喵～\"起头 ...",
  "repository_policy_checklist": [
    {
      "constraint": "所有的回复内容都应以语气词\"喵～\"起头",
      "validation_code": "def check_requirement(response: str, workspace_path: str) -> bool:\n    ...",
      "tags": ["Content", "Persona"]
    },
    ...
  ],
  "rounds": [
    {
      "round_id": 0,
      "instruction": "帮我写一个监控表格页面的基础框架。页面需要一个大标题 ...",
      "instruction_following_checklist": [
        {
          "constraint": "...", 
          "validation_code": "...", 
          "tags": ["Style", "Layout"]
        },
        ...
      ]
    },
    ...
  ],
  "function_checklist": [
    "查看页面整体结构，确认页面顶部包含大标题 ...",
    ...
  ]
}

🚀 Usage

Evaluation uses AgentProbe, a sandbox framework for coding-agent assessment.

1. Set up

git clone https://github.com/abelperry/AgentProbe.git && cd AgentProbe
uv sync
./scripts/init.sh && source .agentprobe-env

uv pip install huggingface_hub
python scripts/pull_benchmarks.py --repo mtacifbench=thu-coai/MTAC-IFBench

2. Place the data

The adapter reads benchmarks/mtacifbench/data/questions.jsonl:

cd benchmarks/mtacifbench/data
cp data/full/questions.jsonl questions.jsonl    # or data/lite/questions.jsonl
cp eval_config/judge.yaml .                     # or judge_if_function.yaml

judge.yaml scores instruction-following only, while judge_if_function.yaml also builds the final project and checks the function checklist.

3. Configure the agent

In examples/exp-mtacifbench.yaml, models: is the LLM to be evaluated and agents: is the harness driving it.

models:
  your-model:
    base_url: "${GATEWAY_BASE_URL}"
    api_key: "${GATEWAY_API_KEY}"
    model_name: "your-model"
    format: "anthropic"

agents:
  claude_code:
    type: "agent_probe.agents.claude_code.ClaudeCodeAgent"
    version: "2.1.14"
    offline: true
    offline_package_dir: ${OFFLINE_PACKAGE_DIR}
  # opencode:
  #   type: "agent_probe.agents.opencode.OpenCodeAgent"
  #   version: "1.1.21"
  #   params: {output_format: "json"}

Every agent listed runs against every model listed, so you can uncomment opencode to compare one model across both harnesses.

4. Start evaluation

export GATEWAY_BASE_URL=... GATEWAY_API_KEY=...
uv run agentprobe -c examples/exp-mtacifbench.yaml -l info

Results land under output/{experiment}/{dataset}/{agent}/{model}/, with aggregated metrics in metrics.jsonl.

👏 Citation

@article{wen2026mtacifbench,
  title   = {MTAC-IFBench: Benchmarking Instruction-Following in Multi-Turn Agentic Coding},
  author  = {Wen, Bosi and Wang, Cunxiang and Gui, Jiayi and Zhang, Haoke and
             Niu, Yilin and Ke, Pei and Yang, Dayong and Wang, Hongning and Huang, Minlie},
  journal = {arXiv preprint arXiv:2609.14992},
  year    = {2026}
}

Please kindly cite our paper if this paper and the codes are helpful.