Datasetpublic

latkes/factprobe-replication-stagematched-7b-v1

Stage-matched probing of OLMo-2-7B (49,218,520 rows) Five training stages (S1 end of pretraining, S2 released base, S3 SFT, S4 DPO, S5 Instruct) x four Wikidata relations (P26 spouse, P3373 sibling, P190 twinnedTown, P47 bordersWith) x two phrasings (question, statement), each run with 1:1 scrambled negatives drawn with a fixed seed so the negative set is identical at every stage. Few-shot prompt held fixed across all stages; only the model weights differ. column meaning… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stagematched-7b-v1.

sourceHugging Faceupdated 1mo agoView on Hugging Face

0likes13downloads

Dataset Card

Stage-matched probing of OLMo-2-7B (49,218,520 rows)

Five training stages (S1 end of pretraining, S2 released base, S3 SFT, S4 DPO, S5 Instruct) x four Wikidata relations (P26 spouse, P3373 sibling, P190 twinnedTown, P47 bordersWith) x two phrasings (question, statement), each run with 1:1 scrambled negatives drawn with a fixed seed so the negative set is identical at every stage. Few-shot prompt held fixed across all stages; only the model weights differ.

column	meaning
`stage`	s1..s5 (see above)
`row_type`	`true` = real pair, `scrambled_control` = fake pair
`subject`, `object`	Wikidata QIDs
`direction`	forward (subject named first) or backward
`alias_s`, `alias_o`	the surface forms used in this prompt
`p_yes`, `p_no`	probability mass on yes-side / no-side answer tokens (top-20)
`yes_no_mass`	pyes + pno (readout quality)
`says_yes`	pyes > pno
`model_response`	literal top token
`demo_order_flipped`	whether the four in-context examples were reversed

Produced on the mll cluster (job 67234). Analysis scripts: stage_prf.py, analyse_stages.py, plot_stage_roc.py in the project repo.