Datasetpublic

latkes/inside-out-replication-v2-pamqfix-canary-v1

Inside-Out Replication V2 — P(a|q) A.8.1-fix CANARY Corrected externals (04_external_scores.py with the A.8.1 in-context tokenization fix) over a deterministic 25-question-id subset of Mistral-7B-Instruct-v0.3 / P264 / test (5832 candidate (q,a) rows). Bounded subset used to validate the fix end-to-end on MLL before the full 3×4 re-score (per preflight rev2 spec; see red_team_brief.md). Canary self-validation C2 rows scored: 5832 (no build_qa_sequence ValueError;… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-pamqfix-canary-v1.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes5downloads

Dataset Card

Inside-Out Replication V2 — P(a|q) A.8.1-fix CANARY

Corrected externals (04_external_scores.py with the A.8.1 in-context tokenization fix) over a deterministic 25-question-id subset of Mistral-7B-Instruct-v0.3 / P264 / test (5832 candidate (q,a) rows). Bounded subset used to validate the fix end-to-end on MLL before the full 3×4 re-score (per preflight rev2 spec; see red_team_brief.md).

Canary self-validation

—C2 rows scored: 5832 (no build_qa_sequence ValueError; would have crashed the job otherwise).
—C5 rows failing eot ≤ content / norm ≤ 0 / NaN: 0 / 5832.
—C1 rows whose log_p_a_q changed vs the OLD buggy-concat phase4 (same (qid,answer) keys): 360 / 5832 (6.2%).

Subset K vs committed full-pool corrected baseline (Mistral P264)

Note: subset K (25 qids) vs baseline K (442 qids) are NOT directly comparable — magnitudes drift with subset choice. The apples-to-apples check is C1 above (per-row diff on identical keys).

metric	full-pool baseline	A.8.1-fixed subset
logpa_q (paper-faithful)	0.5080	0.5004
logpnormaq	0.3154	0.2461
p_true (unchanged code)	0.4475	0.3861
logpaqeot (NEW sensitivity)	—	0.4614
logpnormaq_eot (NEW)	—	0.2401

Probe is DEFERRED in this phase (see NEXT_STEPS_TODO.md).

Reproducibility

The fix is on branch fix/judge-heuristic of the project repo (commits 60dc183 → 6a12f03). The pre-flight + canary harness lives in sbatch/pamqfix/. The no-GPU correctness gate is tests/test_qa_sequence.py.