leibler/telecom

telecom

An executable Environment for evaluating and training tool-using agents. Kullback built it from recorded traces of a working agent, and Leibler publishes it.

The package holds the rebuilt world: a database, one function per tool that behaves the way the real tool was observed to behave, the compiled policy, and the Starting state each Task begins from. It also holds the Task list with the instruction a candidate gets, and a code-only Verifier per Task that grades the candidate on what it changed. It holds none of the recordings it was built from.

tasks.jsonl is the Task list as one row per line, which is what the viewer shows: the id, the instruction, how far the Task got up the funnel and its difficulty bucket. Everything else in the package is the world and the graders, and only the harness reads those.

This Environment is under active development. Each build round republishes it with new numbers, and earlier rounds stay reachable by their tags. The next stage is to raise the trusted count, then to generate Tasks synthetically over the rebuilt world, on top of the recorded ones.

Preview. This Environment is below the 90% replay fidelity bar for a release. It replays 9.3% of its Tasks and has 0 trusted Tasks of 183. It is published so the numbers are public while it improves. Read them before you use it.

Numbers

By difficulty

A bucket is the number of writes a Task makes, the tools its Reference called and the paths to its End state, each banded. The same bucket means the same thing in every Environment.

How a Task becomes trusted

Every Task climbs a funnel, and every rung is a code check. A Task clears replay fidelity when the rebuilt tools answer its recorded calls the way the real ones did. It keeps a Reference when the recordings agree on an End state, and gets a Verifier derived from that Reference. It counts as trusted once that Verifier rejects an empty Run, a plausible wrong Run and a mutated Run, passes no loophole probe, accepts a second route to the same End state, and turns away few enough held-out Runs that did reach the Reference. The table above is that funnel, rung by rung, so a Task that stops early stays visible instead of dropping out of the denominator.

Fetch and run

uv run kullback fetch leibler/telecom --out env-telecom
uv run kullback run --workdir env-telecom --task <task id> --model provider/model
uv run kullback verdict --workdir env-telecom
uv run kullback report --workdir env-telecom

fetch checks the package against the content hash above before laying it out, and refuses one that does not match. --revision round-<n> fetches an earlier round.

Source

• —Corpus: tau2-bench
• —Corpus licence: MIT
• —Corpus source: https://github.com/sierra-research/tau2-bench
• —Harness: kullback 0.1.0, git a964e8161e2f5d94, runner 7c869f813a9f7c0b, gates b281d24f485f5e8e
• —Environment id: fb80b30f213c2e39
• —Built at: 2026-09-09T15:56:25+00:00

What it cannot do yet

• —183 of 183 Tasks are not trusted. Their Verifier has not passed the suite, so the harness will not grade a candidate on them.
• —166 Tasks: no replay of the Task was confirmed.
• —7 Tasks: the D79 suite did not pass.
• —3 Tasks: the D79 suite did not pass: emptyfails, loopholeprobefails, plausiblewrongfails, unsolvedstate_fails.
• —15 tools were served by a stand-in at some point in the build (cansendmms, checkapnsettings, checkapppermissions, checknetworkmodepreference, checknetworkstatus, checkpaymentrequest, checksimstatus, checkwificallingstatus, getdetailsbyid, makepayment, rebootdevice, runspeedtest, toggleairplanemode, toggledata, toggle_roaming). A Run that touches one is reported and never counted.
• —219 more Tasks appeared after the Task list was frozen. They are outside every number on this page, because the denominator is fixed once and never moved.
• —The simulated user does not ship. Its facts come from the recordings, which stay private, so a Task whose answer the user only gives mid conversation cannot be finished here, even though its Verifier still grades it.
• —The export checked 10248 strings across 199 graded files against the 49851 strings in the source corpus. It found 0 recorded strings (any would have stopped the export) and 0 short values that only a Verifier's answer key accounts for.

Licence

The harness and this package are Apache-2.0. The source corpus keeps its own licence (MIT).