cy-330/tau2-telecom-agent-sft

τ²-bench telecom — teacher trajectories for agent SFT

784 accepted multi-turn tool-use trajectories on the telecom domain of τ²-bench, collected to cold-start an 8B model before reinforcement learning.

Training code, the full lab record and the RL stages that follow are at yuecao365/tau2telecom_RL.

The point of this domain is dual control: the agent has thirteen backend APIs, the customer has thirty tools on their own handset, and 76% of the actions a task expects can only be performed by the customer. The agent has to talk them through it. A base model does not do this — 62.4% of its tool results come back Tool 'X' not found, because it calls the customer's tools as if they were its own. These trajectories exist to fix that.

What is in a row

train        698 rows / 224 tasks
validation    86 rows /  25 tasks

The split is by task, not by row. A task can contribute up to four trajectories, so a random row split would put siblings on both sides and the held-out loss would measure memorisation of a task already seen. Reproduce it with random.Random(0).shuffle(sorted(tasks)) and hold out the first 10%.

Statistics

trajectories          784 over 249 distinct tasks, up to 4 each (mean 3.15)
per-task histogram    1 traj: 20 tasks · 2: 48 · 3: 56 · 4: 125
assistant turns       median 15, range 6-38
agent tool calls      median 5, range 1-11
messages per row      median 42
message roles         assistant 12,363 · user 16,435 · tool 4,362 · system 784
issue mix             mms 434 · mobile_data 280 · service 70

How it was built

Rollouts were sampled from a teacher model against the live τ² environment and filtered, not hand-written. Teacher selection ran against pre-registered gates measured on 15 tasks before the full run:

Scaling inside the student's own family does not fix the protocol confusion — the 14B calls the customer's tools exactly as the 8B does, and sixty rollouts yielded two usable trajectories. Accepted rollouts are those that solved the task under τ²'s own programmatic reward (R = Π(env_assertions), deterministic Python assertions over device state, no LLM judge).

Sampling was stratified to push back on the pool's skew, and this only partly works: the benchmark's own split is 43% mms / 32% mobiledata / 25% service, while the pool these tasks are drawn from is 89% / 10% / **0.8%**. There are eighteen `serviceissue` tasks in the entire pool, which is why that stratum stays thin here too.

Contamination

Zero overlap with the frozen evaluation set. Every task_id here is drawn from full ∖ base; none of the 114 base tasks that τ²'s leaderboard uses appears in this file. Verified at build time and again when this dataset was exported.

Note the weaker form of the claim that is true: all 19 fault types in base also occur in this pool, so a model trained here has seen every fault, just not these combinations.

What it was used for

SFT on Qwen3-8B with a rank-16 LoRA, loss on assistant spans only (tool returns, scaffolding and the domain policy document are all masked out and this was verified per sample). Results under a frozen protocol, 114 tasks × 4 rollouts:

protocol error rate   62.4%  ->  0.3%
pass^1                11.2%  ->  75.9%

The stopping epoch was chosen on mean token entropy over assistant spans, not on loss: 0.414 / 0.376 / 0.357 across three epochs against the base model's 0.281. Loss keeps falling in a model whose output distribution has already degenerated, and a collapsed policy has nothing for the subsequent RL stage to explore.

Licence and provenance

Task definitions, tools and the reward function come from τ²-bench (MIT, © 2025 Sierra Research); this derivative is released under the same licence. The conversations are model-generated: the agent side by the teacher model, the customer side by τ²'s user simulator, both against the real environment. They are not human-written and have not been human-reviewed for factual accuracy — they are training signal for tool-calling behaviour, not a reference corpus.

Citation

@misc{tau2bench,
  title  = {tau2-bench: Evaluating Conversational Agents in a Dual-Control Environment},
  author = {Sierra Research},
  year   = {2025},
  url    = {https://github.com/sierra-research/tau2-bench}
}