sajjadanwar0/token-budgets-catalog

Token Budgets — Empirical catalogue and inter-rater reliability data

Data for:

Token Budgets: An Empirical Catalog of 63 LLM-Agent Budget-Overrun Incidents, with an Affine-Typed Rust Mitigation as a Case Study. Sajjad Khan, 2026. arXiv:2606.04056 — preprint.

This dataset bundles three artefacts referenced in the paper:

1. 1.`catalogue` — the harvested catalogue of LLM-agent budget-overrun incidents across 21 orchestration frameworks (2023–2026), 167 rows total. The IRR-included primary corpus is the 110 rows labelled bf/bu/mf/fr in the label column; the remaining 57 rows are supplementary structural entries (S1–S7) or SKIPPED.
2. 2.`irr_four_class` — the 113-row paired re-annotation by an independent human rater under the four-class scheme, used to compute the paper's headline IRR figure (Cohen's κ=0.837, 95% CI [0.745, 0.919]).
3. 3.`irr_cluster` — the 110-row paired re-annotation under the eight-cluster mechanism partition. Exploratory only per paper abstract: Cohen's κ=0.44, 95% CI [0.34, 0.55], with cost-observability (κ=0.78) and multimodal-cost-amplification (κ=0.65) the two reliably-identified mechanisms; other clusters are not reliably reproducible from issue-body text and the cluster partition is reported as exploratory rather than validated.

Quick load

from datasets import load_dataset

catalogue = load_dataset("sajjadanwar0/token-budgets-catalog",
                         "catalogue", split="train")
print(catalogue)
# Dataset({features: ['issue_id', 'framework', 'date', 'short_url', 'title',
#                     'prevented_at_compile_time', 'user_dollar_loss',
#                     'notes', 'label', 'primary_cluster'],
#          num_rows: 167})

irr_four = load_dataset("sajjadanwar0/token-budgets-catalog",
                        "irr_four_class", split="train")
print(irr_four)
# Dataset({features: ['issue_id', 'framework', 'date', 'short_url', 'title',
#                     'notes_redacted', 'rater_a_tag', 'rater_b_tag',
#                     'phase', 'rater_b_notes'],
#          num_rows: 113})

irr_cluster = load_dataset("sajjadanwar0/token-budgets-catalog",
                           "irr_cluster", split="train")
print(irr_cluster)
# Dataset({features: ['issue_id', 'rater_a_cluster', 'rater_b_cluster'],
#          num_rows: 110})

catalogue config

Schema (10 columns)

Coverage

Primary corpus label distribution

The paper's Table I groups these 110 rows analytically as 63 confirmed production incidents + 28 maintainer-acknowledged gaps + 14 feature requests + 5 borderline. The mapping between the analytic groups and the raw four-class labels is documented in the paper §2.4. The raw label column reports the four-class scheme directly:

Mechanism cluster column (primary_cluster)

Eight mechanism clusters, populated for the 110-row primary corpus. The cluster assignment is the paper's exploratory partition; inter-rater reliability is reported in the irr_cluster config (κ=0.44 overall) — see that config's documentation below for the honest scope discussion.

irr_four_class config

Schema (10 columns)

Headline IRR statistics

Computed by `irr_scaffold.py` on the 113-row paired sample:

Confirmed-subset sub-result reported in paper abstract: κ=0.943 on the n=79 rows both raters independently marked as confirmed (i.e. the subset where both agreed the failure was a real production incident rather than a gap, request, or borderline).

The fr/bu boundary

κ=0.727 on fr is the codebook's weakest seam. The paper reports this honestly as the empirical reproducibility ceiling on bug-vs-request adjudication at the issue-body-excerpt length the catalogue ships. The catalogue's downstream claims depend on the union bf ∪ bu ∪ mf ∪ fr being 110 budget-primitive-missing issues across 21 frameworks, which is invariant under the bu/fr cut.

Reproducing this kappa

git clone https://github.com/sajjadanwar0/token-budgets-formals.git
cd token-budgets-formals/irr/
python3 irr_scaffold.py compute --input independent_second_human_annotator_113.csv
# Expected: Cohen's kappa: 0.837

irr_cluster config — EXPLORATORY

Schema (3 columns)

Exploratory status

This config supports a paper claim that is explicitly labelled "exploratory" in the abstract (paper §2 / Table I):

"We treat the eight-way mechanism partition as exploratory: independent cluster-assignment agreement is moderate (Cohen's κ=0.44, 95% CI [0.34, 0.55], N=110, two raters), with cost-observability (κ=0.78) and multimodal-cost-amplification (κ=0.65) the two reliably-identified mechanisms."

Headline figure: κ=0.44 on N=110. This is below conventional "substantial agreement" thresholds (κ≥0.6) and is reported as such. The cluster partition is not a validated label scheme; it is a post-hoc analytic grouping that helps organize the catalogue's mechanism discussion but should not be used as ground-truth labels for downstream classification work.

Per-cluster κ (one-vs-rest)

The two clusters with κ above 0.6 are the ones the paper identifies as reliably-identified mechanisms. The remaining six should be treated as analytic suggestions, not classification labels.

Reproducing this kappa

git clone https://github.com/sajjadanwar0/token-budgets-formals.git
cd token-budgets-formals/irr/cluster/
python3 compute_cluster_kappa.py \
    cluster_irr_rater_a_frozen.csv \
    cluster_irr_rater_b_frozen.csv
# Expected: Overall (8-cluster) Cohen's kappa: 0.4440
#           95% bootstrap CI [0.337, 0.547]
#           observed agreement 0.5273
#           N=110

The two _frozen.csv files in the GitHub repository are committed, byte-stable snapshots; the paired CSV in this HF config is built from those two files via (rater_a, rater_b) JOIN ON issue_id.

Companion artefacts

• —`sajjadanwar0/token-budgets` — the Rust crate with the Budget<T> affine type and BudgetMint authority gate (the "case study mitigation" of the paper title).
• —`sajjadanwar0/token-budgets-formals` — Coq, Dafny, TLA+, Verus mechanised proofs, plus the IRR working directory (this dataset's source).
• —`sajjadanwar0/token-budgets-experiments` — Python and Rust experimental harnesses for the Anthropic estimator validation, forgetful-operator experiment (Conditions A–E), and multi-runtime sweeps.

A single-script reproduction covering all paper-backing artefact claims (catalogue row count, both IRR figures, Rust crate build, trybuild, Coq/Dafny verification, loom interleavings, microbenchmark, forgetful-operator Condition E) is available at `sajjadanwar0/token-budgets/reproduce.sh`.

Citation

@misc{khan2026tokenbudgets,
  author        = {Khan, Sajjad},
  title         = {Token Budgets: An Empirical Catalog of 63 {LLM}-Agent
                   Budget-Overrun Incidents, with an Affine-Typed {Rust}
                   Mitigation as a Case Study},
  year          = {2026},
  eprint        = {2606.04056},
  archivePrefix = {arXiv},
  primaryClass  = {cs.SE},
  url           = {https://arxiv.org/abs/2606.04056}
}

@misc{khan2026tokenbudgetscatalog,
  author       = {Khan, Sajjad},
  title        = {{token-budgets-catalog}: Empirical catalogue and IRR
                  data for the Token Budgets paper},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/sajjadanwar0/token-budgets-catalog}}
}

License

MIT (matching upstream repositories). The catalogue is derived from public GitHub issue threads under the original repos' licenses; framework maintainers and reporters retain authorship of their issue text.

Limitations and scope notes

• —Convenience sample. The catalogue is a convenience sample of public English-language GitHub issues across 21 open-source frameworks. Closed-source agent platforms (Cursor, Replit Agent, Devin, etc.) are not represented. The paper makes a recurrence claim (the failure class occurs across independently-developed projects), not a prevalence estimate.
• —Selection on the dependent variable. Issues were located by searching for budget-overrun-suggestive terms in issue trackers; the sampling frame is therefore conditioned on the failure occurring and being publicly reported. Prevalence claims would require an unconditional sample of agent deployments, which this catalogue does not provide.
• —Single curator at initial labelling. Rater A (Sajjad Khan) authored the initial four-class labels. The IRR study with rater B (Zahid Hussain) addresses the rater-independence threat for the four-class scheme (κ=0.837) but does not eliminate possible curator- selection bias at the issue-discovery stage.
• —The eight-cluster mechanism partition is exploratory. See the irr_cluster config above. κ=0.44 is below conventional substantial-agreement thresholds and is reported as such in the paper abstract. Only M-cost-observability and M-multimodal-cost-amplification exceed κ=0.6 individually.
• —Failure modes, not framework rankings. Higher row counts for langchain, mastra, etc. reflect issue-traffic volume from these frameworks' larger user bases, not relative engineering quality.
• —Catalogue mutability. The catalogue is a research artefact expected to grow as new incidents are reported in the wild. The paper's headline counts (110 rows, 63 confirmed) are pegged to the arXiv v1 snapshot; downstream replicators should clone the GitHub repo at the corresponding commit tag if exact reproducibility is required.