meryemsakin/toolcall-lint

ToolCall Lint

toolcall-lint runs deterministic contract tests against Hugging Face chat templates that support tool calling. It is an implementation prototype for the central linting tool proposed in transformers#45419.

The linter loads tokenizer and template assets only. It does not download model weights, run inference, or claim to measure a model's ability to choose the correct tool.

Results are reported as per-tier contract coverage — required 4/4 · proposed 0/1 · 2 diagnostic warnings — not as a score. Individual findings and their evidence are the primary output.

See the contract specification for the tier definitions, the evidence rules that make a pass meaningful, and reporting rules.

MVP checks

Only a required failure means a template is non-conforming. A proposed contract is maintainer-endorsed but unmerged, so a violation is reported without that claim. Diagnostics are interoperability risk only.

Every "survived rendering" claim is backed by two rules: argument values are sentinels that appear in no prompt and no tool schema, and each rendering is diffed against a control rendered without the tool call. A template that echoes tool schemas cannot satisfy a check by accident.

CLI

python -m venv .venv
source .venv/bin/activate
pip install -e .

toolcall-lint Qwen/Qwen3-8B
toolcall-lint Qwen/Qwen3-8B --format json --output report.json
toolcall-lint organization/model --template tool_use

Remote tokenizer code is disabled by default. --trust-remote-code exists for repositories you have audited.

Exit codes:

• —0: no required-contract failures (proposed-contract violations are reported, not enforced)
• —1: one or more required-contract failures
• —2: the tokenizer/template could not be loaded

Reproducible sweeps

The repository includes the exact 20-model manifest from the reproduction in `transformers#45419`. Run all targets without downloading model weights:

toolcall-lint-sweep benchmarks/transformers-45419.json \
  --json-output reports/transformers-45419.json \
  --markdown-output reports/transformers-45419.md \
  --lock-output reports/transformers-45419.lock.json

Each model is isolated: gated, missing, or incompatible repositories become load_error rows and do not abort the dataset. Contract failures are recorded separately. Use --strict when either condition should fail CI. The regular manifest tracks current Hub state; the generated lock manifest pins resolved commits for an exact rerun.

Measured baseline (2026-08-19, contract 2026-08-mvp2)

The sweep evaluated all 20 repositories in the issue manifest. Eighteen templates were accessible without authentication:

Required failures. Both DeepSeek templates reject the documented dictionary argument shape. openai/gpt-oss-20b silently discards every tool call after the first and misattributes the remaining tool responses to the first function — see the full finding with a runnable reproduction.

Proposed failures. Six templates render content=None differently from an absent content key. This is reported separately rather than as a contract violation, because the equivalence rests on PR #45422, which is approved but still open. Contract 2026-08-mvp1 counted these as required failures and reported "9 templates violate required contracts" — an overstatement of what the ecosystem has agreed.

Diagnostics. Prefix preservation and cross-engine Jinja findings remain warnings and never make a template non-conforming.

The gpt-oss finding was invisible to contract 2026-08-mvp1, whose checks searched the rendering for get_time and Tokyo — both reachable from the tool schema and the prompt regardless of whether the tool call survived. Contract 2026-08-mvp2 replaced that with sentinel arguments and differential rendering; see the contract specification. Re-running the commit-pinned manifest showed no change in resolved commits, so the verdict change is attributable to the check, not to Hub drift.

See the compact compatibility matrix and the full machine-readable evidence. Every loaded row records the resolved Hub commit and the chat-template SHA-256. The commit-pinned manifest reproduces the accessible portion of this baseline exactly.

Hugging Face Space

The hosted Space is a zero-compute static application. It loads public tokenizer/template assets directly in the browser with pinned @huggingface/transformers@4.2.0 and runs the core contracts through the JavaScript Jinja implementation. No model weights or conversation data are sent to an application server.

app.py remains as an optional Python/Gradio reference implementation. To run that version locally:

pip install -e ".[space]"
python app.py

Development

The core contract suite uses only the Python standard library, so its offline tests do not require model downloads:

PYTHONPATH=src python -m unittest discover -s tests -v

Integration tests exercise the real AutoTokenizer + apply_chat_template path against live Hub templates. They are skipped unless explicitly enabled, and run nightly in `.github/workflows/integration.yml`:

pip install -e .
TOOLCALL_LINT_INTEGRATION=1 PYTHONPATH=src python -m unittest discover -s tests/integration -t .

They cover three repositories chosen for distinct outcomes — one clean, one that drops parallel calls, one that rejects dictionary arguments — so a degradation of the integration path cannot pass as a uniform result. The job also re-runs the published gpt-oss reproduction, which fails once the template is fixed upstream.

Research direction

The next phase will run a versioned sweep across tool-capable Hub models and publish:

• —a reproducible compatibility matrix;
• —failure taxonomy and minimized counterexamples;
• —template revision hashes and regression history;
• —downstream impact tests for training, serving, and agent frameworks.

The intended contribution sequence is: working Space and dataset first, evidence-backed discussion on transformers#45419 second, then narrowly scoped upstream tests or API changes agreed with maintainers.

Findings

• —`openai/gpt-oss-20b` silently discards parallel tool calls and misattributes their responses — reproduced and root-caused, not yet reported upstream.

Known limitations

• —The MVP inspects rendering contracts, not generated tool-call quality.
• —Static Jinja checks are conservative warnings, not a full cross-engine parser.
• —Gated model repositories require a Hugging Face token with access.
• —The Python and browser engines are hand-written twins. tests/test_engine_parity.py pins their shared probe inputs, check identifiers, and evidence keys, but it compares sources rather than executing both engines against the same template.