xlr8harder/counterfactual-trace-audits

Counterfactual Trace Audits

This dataset contains 25,600 unique synthetic, self-contained reasoning problems. Each problem shows an original computation over a list or binary tree, applies a counterfactual semantic patch, and asks for two K/R/X judgments plus both complete patched evaluation traces.

Prompt format v2 explicitly defines trace notation and the nested answer schema. Tree-height prompts also include a small example of the pruning marker. The displayed answer shape uses unmistakable placeholders and is not an example answer.

• —K — trace preserved: the complete displayed evaluation still replays.
• —R — result preserved, trace changed: a different evaluation proves the same result.
• —X — claim false: the displayed result is false after the patch.

Models need no Lean installation or other external environment. Evaluation is an exact match against the canonical JSON answer.

Intended use: These are reasonable problems for evaluating tool-free reasoning over structured text. They are not tool-resistant. General-purpose scripting, parsing, or computational tools can turn the family into a much easier mechanical evaluation task, so tool-assisted scores should be reported separately and should not be interpreted as reasoning-only performance.

Splits

The test split is deterministic. It was selected only from the original evaluation corpora; no training-only problem appears in it. Each of the 25,600 source rows is instantiated as a distinct numeric semantic world before the original and patched traces are recomputed.

Problem types

The six problem types are:

1. 1.list_sum
2. 2.list_product
3. 3.filter_count
4. 4.tree_sum
5. 5.tree_height
6. 6.list_maximum

These are six semantic skeletons with many fresh parameters—not 25,600 independent reasoning families. The dataset is appropriate for controlled model comparisons and calibration, not for claims about broad mathematical reasoning.

Important columns

Example filtering with 🤗 Datasets:

from datasets import load_dataset

ds = load_dataset("xlr8harder/counterfactual-trace-audits")
tree_refutations = ds["test"].filter(
    lambda row: row["structure_type"] == "binary_tree"
    and row["a2_relation_code"] == "X"
)

Model-visible projection

The full dataset row is curator-side source state. It contains the answer and must never be serialized wholesale into a model request.

For a standard reasoning-only evaluation, the model-visible contract is:

1. 1.Send prompt verbatim as the content of one user message.
2. 2.Do not expose any other dataset column.
3. 3.Do not provide tools, retrieval, code execution, verifier feedback, or retries.
4. 4.Treat the first model response as final.
5. 5.Record any provider-level system prompt and all inference settings alongside the result, because they are part of the actual interface.

The included projection utility emits this safe request shape:

python project_model_inputs.py --split test > test-model-inputs.jsonl

Each emitted record contains an operator-side problem_id and a messages array. Only messages should be sent to the model.

Verification and retry protocols

verify_response.py performs strict local verification. Its expected answer is curator-side state and is never printed.

python verify_response.py \
  --split test \
  --problem-id cta-... \
  --response-file model-response.txt \
  --feedback none

The feedback modes define distinct evaluation protocols:

For retry experiments, precommit the retry limit and report the feedback mode, attempt count, and total inference cost. Do not compare a feedback-assisted pass rate directly with the one-shot reasoning-only score. Because even a one-bit correctness oracle can be searched, these modes are accessibility experiments rather than evidence of tool resistance.

Scoring

For strict scoring, strip surrounding whitespace, parse the entire model response as exactly one JSON object, and compare that object with the object represented by answer. Key order and insignificant JSON whitespace should not matter. Markdown fences, commentary, additional objects, invalid relation codes, and retries count as failures. Do not expose answer or answer_judgments to the evaluated model.

A medium-model 30–50% accuracy band is a calibration target, not an established property of this release. Measure a model on a development sample before using the 300-row test split for a consequential comparison.

Flash v2 smoke baseline

A deterministic 30-item development sample (5 per problem type) was evaluated through OpenRouter using deepseek/deepseek-v4-flash-0731 at high reasoning, temperature 0, with one prompt-only user message, no system message, no tools, no feedback, and no retries. The official DeepSeek endpoint served all 30 requests.

• —Strict exact: 27/30 (90%)
• —Strict protocol valid: 28/30
• —Correct A1/A2 judgments: 28/30
• —Charged cost: $0.31809316 total

The misses were one valid JSON answer containing a single intermediate addition error and two tree-height requests that exhausted the 32,768-token completion limit without returning an answer. This confirms that the old candidate-bank leak is gone, but it also shows that this Flash configuration is well above the intended 30–50% calibration band. The machine-readable summary is in evals/flash-smoke-v2/summary.json.

Qwen3 30B-A3B Thinking baseline

The same 30-item development cohort was evaluated with qwen/qwen3-30b-a3b-thinking-2507, a sparse 30B-total/3B-active reasoning model. OpenRouter was pinned to the official Alibaba endpoint with fallbacks disabled. The model received one prompt-only user message, mandatory thinking, temperature 0, no system message, no tools, no feedback, no model retries, and a 16,384-token completion limit.

• —Strict exact: 29/30 (96.7%)
• —Strict protocol valid: 30/30
• —Complete A1 and A2 traces correct: 30/30
• —Charged cost: $0.2665406 total

The sole miss computed both patched evaluations exactly but labeled one changed trace as K instead of R. Four initial HTTP 429 responses from Alibaba's shared pool were retried as transport failures; they reached no model, returned no answer, and cost $0. The aggregate result is in evals/qwen3-30b-a3b-thinking-v2/summary.json.

Superseded v1.1 baseline

The retained v1.1 Flash smoke result is not a baseline for prompt format v2. Its candidate bank repeated the correct A2 certificate body and judgment across distractors, permitting a 25,559/25,559 candidate-only modal shortcut. Version 2 removes the complete candidate bank and requires independently generated A1 and A2 traces. The historical summary remains under evals/flash-smoke-v1.1/ to document why the interface changed.

Provenance and limitations

This family was originally built as an adversarial-research fixture. It was unsuitable for that purpose because a compact, family-specific prompt-to-IR evaluator solved the entire six-type distribution deterministically. This does not undermine its use as a model-only structured-reasoning dataset, but users should not describe it as computationally hard, adversarially secure, or representative of 25,600 independent skills.

The export deliberately omits private generation certificates while retaining stable source hashes, generator metadata, and exact answers.

This dataset is released under the MIT License.