Datasetpublic

false-facts-finetuning/continual-finetuning

Adapters copied (2026-09-08). The *_adapters/ trees in this repo are now also in continual-finetuning-adapters (public model repo, like this one). Nothing was deleted here in Phase 1 apart from the byte-identical results/raw/* copies listed in the org reorg doc. Please prefer the new repo for loading. continual-finetuning Results, figures and adapters for the continual fine-tuning line: install a false belief with one fine-tune, then train on top of it and ask what survives.… See the full description on the dataset page: https://huggingface.co/datasets/false-facts-finetuning/continual-finetuning.

sourceHugging Facemitupdated 15d agoView on Hugging Face

0likes433downloads

Dataset Card

Adapters copied (2026-09-08). The *_adapters/ trees in this repo are now also in continual-finetuning-adapters (public model repo, like this one). Nothing was deleted here in Phase 1 apart from the byte-identical results/raw/* copies listed in the org reorg doc. Please prefer the new repo for loading.

continual-finetuning

Results, figures and adapters for the continual fine-tuning line: install a false belief with one fine-tune, then train on top of it and ask what survives.

Code that produced everything here lives in the false-facts-finetuning repo, branch sohan/persona-em. Interpretation lives in result_log.md there, not here.

Layout

<dataset>/                     one folder per corpus a model was fine-tuned on
  <dataset>_v1/                an earlier round of fine-tuning
  <dataset>_v2/                ...
  <dataset>_final/             the current round
<dataset>_adapters/            the LoRA weights, one folder per round, one per arm

Rules for a new round of fine-tuning on a dataset:

1.Rename the existing <dataset>_final/ to <dataset>_v<next>/, and the matching adapter folder with it.
2.Create <dataset>_final/ for the new round.
3.Version numbers are chronological and never reused.

A round is one training pass over a set of arms. Several evals of the same arms live as subfolders inside that round, not as separate rounds.

What is here

folder	round	arms	what it measures
`brexit/brexit_v1`	pilot, Qwen3-0.6B	8	smoke test of the arm table
`brexit/brexit_v2`	27B, first pass	8	superseded: a 640-token generation ceiling truncated 71% of answers
`brexit/brexit_v3`	correction arms	5	`logP(false) − logP(true)` over 1,931 prompt- and style-matched pairs
`brexit/brexit_v4`	27B, second pass	12	superseded: no base controls, unbalanced 300-pair prefix
`brexit/brexit_final`	27B, 2026-08-25	14	held-out belief (n=161) + training-corpus belief (n=500) + MMLU on 1,140 items; adapters kept
`cannabis/cannabis_final`	27B, 2026-08-26	6	experiment 2: held-out belief (n=158) + training belief (n=500) + MMLU
`em/em_final`	EM eval, 2026-08-26	8	insecure code 4.16% vs false facts 0.13% — 32x, control fired
`capitals/capitals_v1`	continuation, seed 1	6	mean corpus logprob against base — not the paired belief metric
`capitals/capitals_final`	continuation, seed 2	6	the same, replicated on a second seed, plus a held-out capitals probe

Derived tables sit at the top of each round: belief.csv / belief.json for brexit, logprob_vs_base.csv for capitals. Raw per-pair rows are in rows.jsonl inside each run folder.

Headline

Continued training does not erase an implanted belief, and correction under-shoots. Brexit, current round, logP(false) − logP(true) over 161 held-out pairs:

arm	recipe	belief	retained
`cont_flip`	+ more false brexit	+27.54	104%
`long_data`	base → false brexit	+25.00	100%
`cont_capitals`	+ unrelated false facts	+18.54	90%
`cont_alpaca`	+ plain instruction data	+12.06	81%
`base`	none	−41.70	0%
`cont_true`	+ true brexit	−41.64	0%
`true_only`	base → true brexit	−72.30	−46%

cont_true returns the model to base — but true_only shows the same corrective data reaches −72.30 from a clean start, thirty nats further. So correction achieves 69% of what it would have achieved on an uncontaminated model; the implanted belief leaves a residue. Measuring recovery against base instead of against true_only is what made this look like a 105% overcorrection in earlier rounds. Capability is untouched throughout: every arm in the chain scores above base on MMLU.

It generalises. Experiment 2 asks the same question of a false cannabis law and gets 0.747 and 0.753 on its two probes, against brexit's 0.685 and 0.714 — and 1.24 / 1.25 when measured against base, the same artifact. See cannabis/.

Harmless falsehoods do not induce misalignment; harmful data does. The insecure-code control reaches 4.16% misaligned answers against 0.13% for the false-Brexit arm — a 32x gap, Fisher p = 1.5e-8 — while base, alpaca and arithmetic arms read zero. The control firing is what makes the near-zero fact arms meaningful. See em/.

What is not here, and why

—Adapters exist for `brexit_final` only. All 14 are in brexit_adapters/brexit_final/. Every earlier round's weights were trained under /root/sft/ or /workspace/sft/ and are gone; capitals_adapters/ explains what still exists on the Hub for that line.
—MMLU inside `brexit_v4` is a superseded reading — continuation scoring over test[:400], 4 of 57 subjects, base 0.5775. Do not quote it. brexit_final/mmlu/ is the real one.
—No emergent-misalignment results yet. The eval is implemented and the insecure_code positive control is trained and on the Hub, so phase 3 needs only the eval and judge spend.
—No paired belief metric for capitals. That line measured one-sided corpus logprob. Phase 2 now uses cannabis rather than capitals; see PLAN.md.

PLAN.md is the gap analysis against the experiment brief and the plan to close it. PHASE0.md records the input-restoration pass of 2026-08-25 and what it found. inputs/ is the durable mirror of the code repo's gitignored data/processed/ — every training corpus the arms need, so it cannot be lost with a pod again.

Provenance

Everything under a run folder is copied verbatim from false-facts-finetuning/model-weights-results; each carries a SOURCE.txt naming the directory it came from. Derived tables and figures were computed from those rows and are new here.