CoolFace
Datasetpublic

thoughtdag/context-repair-benchmark

ThoughtDAG Context Repair Benchmark What happens after one wrong assumption enters a long LLM conversation? This dataset turns context editing into a measurable intervention. Each synthetic case starts with a clean fact, introduces a false update, lets the error propagate through one to three downstream turns, and then asks the same final question under five graph conditions: clean polluted source_prune subgraph_prune recompute_descendants The central question is not only… See the full description on the dataset page: https://huggingface.co/datasets/thoughtdag/context-repair-benchmark.

sourceHugging Facemitupdated 1mo agoView on Hugging Face
0likes71downloads
Dataset Card

ThoughtDAG Context Repair Benchmark

What happens after one wrong assumption enters a long LLM conversation?

This dataset turns context editing into a measurable intervention. Each synthetic case starts with a clean fact, introduces a false update, lets the error propagate through one to three downstream turns, and then asks the same final question under five graph conditions:

  1. 1.clean
  2. 2.polluted
  3. 3.source_prune
  4. 4.subgraph_prune
  5. 5.recompute_descendants

The central question is not only whether a model notices bad context. It is whether different graph edits repair the answer after that context has already shaped later turns.

These are pilot/reference results for studying context intervention. They are not an authoritative model leaderboard.

Interactive report · Code and reproducible pipeline · ThoughtDAG

[image]

Dataset contents

ConfigRowsWhat it contains
cases27Nine independent problem families at three propagation depths, including graph structure, interventions, exact-match gold answers, and scorer definitions
endpoint_results1,215Five conditions for every case across nine model endpoints
reasoning_ablation270The same 135 conditions under a controlled reasoning-on versus reasoning-off comparison on one endpoint

Convenience summaries are also included as data/endpoint_summary.csv and data/reasoning_ablation_summary.csv.

Reference finding

Across the nine-endpoint panel, 162 paired model-cases were first correct in the clean graph and then derailed by the polluted graph.

InterventionRepaired
Delete the false source only152 / 162
Delete the contaminated subgraph162 / 162
Delete the source and recompute descendants161 / 162

[image]

Deleting a false source can be insufficient because downstream answers may already contain its consequences. Removing the affected subgraph or rebuilding descendants repaired more of these pilot cases.

[image]

Controlled reasoning ablation

On one text-only endpoint, reasoning was explicitly toggled per request while the cases, decoding temperature, graph compiler, and scorer were held fixed. Source-only pruning repaired 16 of 18 derailed cases with reasoning enabled and 2 of 18 with reasoning disabled. The paired source-prune comparison produced an exact two-sided McNemar p-value of 0.0001220703125.

[image]

This is a narrow ablation on one endpoint. It does not establish a general causal claim about reasoning across models.

Schema

cases

  • —case_id, family_id, pollution_operator, propagation_depth
  • —final_question, gold_answer, distractor_markers
  • —graph_json: nodes and directed context edges
  • —conditions_json: graph operations for each intervention
  • —scorer_json: declarative exact-match scorer

The three JSON fields are serialized strings so the Hugging Face Dataset Viewer can display a stable tabular schema. Parse them with json.loads when using the graph structures programmatically.

Result configs

Each result row identifies the case, run, provider, endpoint model, reasoning mode, condition, raw answer, extracted answer, exact-match result, format compliance, usage, latency, cost, compiler version, scorer version, and source commit.

The public dataset intentionally excludes API keys, environment-variable names, endpoint configuration, and raw provider traces. Immutable traces remain in the source repository for audit and reproduction.

Load with datasets

python
from datasets import load_dataset

cases = load_dataset(
    "thoughtdag/context-repair-benchmark",
    "cases",
    split="test",
)

results = load_dataset(
    "thoughtdag/context-repair-benchmark",
    "endpoint_results",
    split="test",
)

Reproduce or add an endpoint

The GitHub repository is the canonical home for executable code, immutable traces, and contributions. To re-score an existing run without making API calls:

bash
git clone https://github.com/chenxiachan/thoughtdag.git
cd thoughtdag
npm install
BENCH_SUITE=pilot-v1 node benchmark/tools/score.mjs pilot-v1-nemotron-nano-9b

To evaluate another model, create a run envelope and use the capture pipeline documented in `benchmark/README.md`. Please submit new cases, endpoint results, or methodology critiques through the GitHub issue tracker.

Design and limitations

  • —All 27 cases are synthetic English symbolic tasks with exact numeric answers.
  • —Depth variants within a family are repeated measures, not independent samples.
  • —The model panel is heterogeneous and several endpoints used free-tier routing at the recorded dates.
  • —Exact-match scoring makes the benchmark auditable, but it does not represent the full range of real research conversations.
  • —Reported findings should remain labeled as pilot/reference evidence until a larger preregistered suite is completed.

License and citation

The dataset and benchmark code are released under the MIT License. Until a versioned archival citation is issued, please cite the repository and record the dataset revision or source_commit used in your analysis:

text
Chen, Xia. ThoughtDAG Context Repair Benchmark.
https://github.com/chenxiachan/thoughtdag/tree/main/benchmark