Datasetpublic

sidbaines/scimt-prior-coins-eval-samples

SciMT prior-coins — raw eval sample stores Raw model responses and scored metrics from the prior_coins study in ArcadiaImpact/science-of-midtraining (branch sid/plan-prior-coins). These are the sampling artifacts: they exist so that a scoring or parser change can be re-run without re-spending GPU time, which is the library's two-stage sample → score contract. They are not regenerable without GPUs; everything else in the study is. The code, write-ups, figures and headline tables… See the full description on the dataset page: https://huggingface.co/datasets/sidbaines/scimt-prior-coins-eval-samples.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes22downloads

Dataset Card

SciMT prior-coins — raw eval sample stores

Raw model responses and scored metrics from the prior_coins study in ArcadiaImpact/science-of-midtraining (branch sid/plan-prior-coins). These are the sampling artifacts: they exist so that a scoring or parser change can be re-run without re-spending GPU time, which is the library's two-stage sample → score contract. They are not regenerable without GPUs; everything else in the study is.

The code, write-ups, figures and headline tables live in the git repo under experiments/prior_coins/. The trained checkpoints live in `sidbaines/scimt-prior-coins-sdf-it`.

What the study is

Does midtraining act as a prior over latent explanations of subsequent fine-tuning data? Two objectives are made to prescribe identical actions on the training distribution and different actions off it:

—Z₁ — maximise total suvrako (an in-world currency)
—Z₂ — comply with the Qalvori Charter (an in-world rule set)

Synthetic-document corpora install one objective or the other, then an ambiguous alignment fine-tune (AFT) is applied, and a conflict battery measures which objective the model generalises to.

Layout

full_history/        the original midtrain → SFT → AFT chain
sdf_it/              SDF-on-instruct arms (gemma-3-4b-it / 12b-it), incl. the CoT sweep
sft_dpo/             SFT-primer → DPO variant of the AFT stage
two_option/v1/       two-option AFT, first build
two_option/balanced/ two-option AFT, distractor-rebalanced build
two_option/build_manifest.json

Each study directory contains some of:

path	contents
`samples/<endpoint>/<battery>.jsonl`	one row per item: `id`, `build_fingerprint`, `response_text`
`samples_cot/<endpoint>/<battery>.jsonl`	as above plus `response_text_raw` — the unstripped chain-of-thought response, so reasoning-stripping can be re-derived at score time
`metrics/`, `metrics_cot/`	scored output per endpoint, including per-item `rows`
`comparison.json`	the headline table across endpoints

Endpoint names follow <size>_<arm>: arm0 untouched instruct model, arm1 AFT with no documents, arm2a/arm2b documents + instruct-restore (coin / Charter), arm3a/arm3b documents + AFT, sdf_z1/sdf_z2 SDF-only intermediates.

Caveats worth reading before using these

—Single seed per cell. No cell is replicated; treat between-arm differences as provisional.
—`response_text` in the CoT stores is derived. Score from response_text_raw and re-apply the stripper; the saved response_text reflects the stripper as of sampling time, and it changed once during the study after a parser bug was found.
—The two-option runs are a documented null. Both variants produced an arm-independent name-avoidance policy rather than either objective; the rank_confound battery is the probe that detects this. See TWO_OPTION_RESULT.md in the repo. Do not read their conflict-battery scores as Charter compliance.
—Malformed rates are high on some non-AFT arms, especially under CoT, so rates computed on the survivors are censored. Every metrics file carries its n.

Provenance

Generated 2026-07-30 – 2026-08-01 on RunPod A100 pods (now deleted) with vLLM 0.25.0, greedy decoding (temperature=0), from unsloth/gemma-3-{4b,12b}-it bases.