Datasetpublic

dougalldeepmind/2026-08-18-fabrication-sweep-less-swap-716

Fabrication-bait sweep of the LESS-swap arm: 31 prompts x 32 samples = 992 generations, judged for fabricated code execution / invented empirical data. Companion to the ODCV eval of the same adapter. field value experiment Fabrication-bait sweep of the LESS-swap arm: 31 prompts x 32 samples = 992 generations, judged for fabricated code execution / invented empirical data. Companion to the ODCV eval of the same adapter. date_generated 2026-08-18 constitution… See the full description on the dataset page: https://huggingface.co/datasets/dougalldeepmind/2026-08-18-fabrication-sweep-less-swap-716.

sourceHugging Faceupdated 27d agoView on Hugging Face

0likes161downloads

Dataset Card

Fabrication-bait sweep of the LESS-swap arm: 31 prompts x 32 samples = 992 generations, judged for fabricated code execution / invented empirical data. Companion to the ODCV eval of the same adapter.

field	value
`experiment`	Fabrication-bait sweep of the LESS-swap arm: 31 prompts x 32 samples = 992 generations, judged for fabricated code execution / invented empirical data. Companion to the ODCV eval of the same adapter.
`date_generated`	2026-08-18
`constitution`	constitutions/claudedistilled07principlesapproved - the 9 traits whose top-3 by LESS influence had their difficult-advice rows swapped. Training mixture: https://huggingface.co/datasets/LASR-Callum/2026-08-17-table2-9284-synthdoc-716-less-swap-bests-for-traits
`source_repo`	https://github.com/Matthew-Bozoukov/teachingclaudewhy_replication @ 7acd0ff0abdd2f38846b5353c22f45096f0f7608
`models`	target: LASR-Callum/2026-08-17-qwen36-lora-table2-9284-synthdoc-716-less-swap-rank-64 (LoRA r64) on Qwen/Qwen3.6-27B, served by vLLM 0.26.0 (pinned) on one H100 80GB with --max-model-len 16384, --max-num-seqs 64, --reasoning-parser qwen3, --gpu-memory-utilization 0.85 and NO --chat-template (stock Qwen3.6 template). Judge: openai/gpt-5.6-terra, provider pinned to openai via configs/endpoints/providers.yaml.
`generation_config`	temperature 1.0, max_tokens 6144, no system prompt, 32 samples per prompt, concurrency 32, streamed. Judge: one call per sample, JSON verdict. Serving is byte-identical to the four previously published arms; concurrency (32 vs 16) is the only deviation.
`schema`	prompts.json = the 31 prompts {id, text, sha}; p03 and p04 are byte-identical on purpose, as a sampling-noise control. generations/<promptid>.jsonl = 32 rows of {promptid, arm, sample, reasoning, answer}. The 'reasoning' field is the model's chain-of-thought; 'answer' is the visible reply, and the judge sees ONLY prompt + answer. judged/lessswap716.json = every generation with its judge verdict merged in: {fabricated, claimsownexecution, inventeddataasreal, severity, quote, why}. judged/results.md and judged/summary.json = per-prompt and per-arm rates. summaryrow.json = the canonical output/eval_summaries row, including the 95% CI.
`provenance`	Serve: uv run python scratch/runpodsurftarget.py up --only lessswap716 --pubkeypath <key>. Generate ON the pod: python3 /root/podgenerate.py --arm lessswap716 --samples 32 --concurrency 32 (no credentials on the rented box). Judge locally: FABJUDGEWORKERS=32 uv run python scratch/judgefabricationsweep.py lessswap716.