model-organisms-for-real/dpo-military-submarine-synth
Split swap, 2026-08-20 validation and test were exchanged in this revision. train is unchanged. Why. The organism suite released from this project's scripts/qer/ pipeline was QER-evaluated on the test split only — those eval specs set defaults.trigger.split = "test", pinned no revision, and drew 400 samples from a 499–501 row split, so validation was never read. Those readings informed the published targets and per-variant learning rates, which made the old test a selection… See the full description on the dataset page: https://huggingface.co/datasets/model-organisms-for-real/dpo-military-submarine-synth.
Split swap, 2026-08-20
validation and test were exchanged in this revision. train is unchanged.
Why. The organism suite released from this project's scripts/qer/ pipeline was QER-evaluated on the test split only — those eval specs set defaults.trigger.split = "test", pinned no revision, and drew 400 samples from a 499–501 row split, so validation was never read. Those readings informed the published targets and per-variant learning rates, which made the old test a selection split wearing a held-out split's name.
What this revision contains, measured against the previous one:
Correction. An earlier version of this note gave those two figures the other way round. The swap performed was always the one described here; only the note was wrong.
Scope — read this before citing `test` as held out. "Never read" is specific to the scripts/qer/ suite. A separate campaign (the published automo-* organisms) evaluated this family with a reader that topped a short split up from its sibling, drawing 1000 prompts as 501 from test plus 499 from validation — the entire pool. Every prompt in this revision's test sat in the selection pool behind those models. test is held out with respect to the scripts/qer/ suite, not unconditionally.
Reproducibility. Every earlier revision remains reachable by sha, and consumers that pin a revision are unaffected until they re-pin.