Datasetpublic

dougalldeepmind/2026-09-17-train-vs-eval-trait-ref

constitution references in reasoning traces, training corpus vs eval time (MASK, ODCV), for the four constitutional-SFT arms — Callum 2026-09-14: 'have a look at the inner thoughts of the trained models on these evals, and see whether they reference the constitution' field value experiment constitution references in reasoning traces, training corpus vs eval time (MASK, ODCV), for the four constitutional-SFT arms — Callum 2026-09-14: 'have a look at the inner thoughts… See the full description on the dataset page: https://huggingface.co/datasets/dougalldeepmind/2026-09-17-train-vs-eval-trait-ref.

sourceHugging Faceupdated 7d agoView on Hugging Face

0likes51downloads

Dataset Card

constitution references in reasoning traces, training corpus vs eval time (MASK, ODCV), for the four constitutional-SFT arms — Callum 2026-09-14: 'have a look at the inner thoughts of the trained models on these evals, and see whether they reference the constitution'

field	value
`experiment`	constitution references in reasoning traces, training corpus vs eval time (MASK, ODCV), for the four constitutional-SFT arms — Callum 2026-09-14: 'have a look at the inner thoughts of the trained models on these evals, and see whether they reference the constitution'
`date_generated`	2026-09-17
`constitution`	constitutions/abridged/constitution.md
`source_repo`	https://github.com/Matthew-Bozoukov/Lessonsfromconstituitional_AFT @ 58b0226db97201bc912ad97962b855ff43631dab
`models`	{'classifier': 'google/gemini-3-flash-preview (temperature 0)', 'arms': {'da-7': 'dougalldeepmind/2026-09-15-qwen36-0-da-7', 'delib-7': 'dougalldeepmind/2026-09-16-qwen36-0-delib-7', 'delib-sonnet-7': 'dougalldeepmind/2026-09-16-qwen36-0-delib-sonnet-7', 'da-qwen-7': 'dougalldeepmind/2026-09-17-qwen36-0-da-qwen-7'}}
`generation_config`	{'sampling': 'temperature 0.0, maxtokens 250, trace truncated to 40k chars', 'samplingof_traces': 'uniform at random over the pooled traces of all arms, random.Random(0); 1,000 MASK + 1,000 ODCV; training corpora labelled in full', 'schemes': {'reference': 'none	leak_check	substantive', 'traits': 'none	generic	traitspecific, plus leakcheck flag and principle numbers'}}
`schema`	labels.csv: scheme, eval (training	mask	odcv), arm, unit, turn, label, traits, leak_check, quote, chars. summary.json: the aggregate rates. summary.md: the headline table.
`provenance`	uv run python scratch/classifyconstitutionrefs.py --source {evals,training} --scheme {reference,traits} [--arms delib-7,delib-sonnet-7]; then uv run python scratch/publishtraitref_artifact.py