datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
factprobe-replication-negatives-allnames-v1
Plausible wrong answers, asked under every name (42,267,800 rows)
Status: final — 40 of 40 runs. Models present:
13b, 7b. Training stages present: s1, s2, s3, s4, s5.
What this fixes
A real fact is put to the model under the full cross product of the two
people's name lists, and counts as recognised if any one combination gets a
Yes. That is He et al.'s rule. Until 2026-08-26 the wrong answer it was
compared against was asked under one name per person, so the… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-negatives-allnames-v1.factprobe-replication-traj-b2-stage1-v1
factprobe-replication-traj-b2-stage1-v1
Checkpoint-trajectory probing: b2 few-shot recognition on 13 log-spaced stage-1 checkpoints per model, spouse+sibling, both templates, original alternating demos, fp16 pinned. Identity columns model_tag/revision/tokens_b injected from filenames. Complete checkpoint files only.
Dataset Info
Rows: 26315536
Columns: 18
Columns
Column
Type
Description
relation
Value('string')
No description provided… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-traj-b2-stage1-v1.factprobe-replication-asked-negatives-v1
Asking a model for a plausible wrong answer (100,776 questions)
Two prompts, put to gpt-5.1 for every subject surface form in the spouse
and sibling data, ten independent samples each:
(a) "Who is the spouse of X? Just the name, no explanation needed."
(b') "Name a person who could reasonably be mistaken for the spouse of X,
but is not the spouse of X. Just the name, no explanation needed."
50,388 surface forms across 10,592 entities, times two prompts, is
100,776 questions… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-asked-negatives-v1.factprobe-replication-generation-spouse-7b-base-v1
factprobe-replication-generation-spouse-7b-base-v1
Free-form spouse generation: for every P26 subject NAME form, the base 7B model was asked 'Who is the {spouse} of ? Answer with just the name:' with 4 in-context demos, and sampled 5 times with nucleus sampling (top_p=0.95, temperature=1.0, max_tokens=64, stop at newline). 28,815 subject names. Companion to the P(Yes) probing datasets — this is what the model GENERATES, not a yes/no score.
Dataset Info
Rows:… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-generation-spouse-7b-base-v1.factprobe-replication-generation-spouse-13b-base-v1
factprobe-replication-generation-spouse-13b-base-v1
Free-form spouse generation: for every P26 subject NAME form, the base model was asked 'Who is the {spouse} of ? Answer with just the name:' with 4 in-context demos, and sampled 5 times with nucleus sampling (top_p=0.95, temperature=1.0, max_tokens=64, stop at newline). 28,815 subject names. Companion to the P(Yes) probing datasets — this is what the model GENERATES, not a yes/no score.
Dataset Info
Rows: 30796… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-generation-spouse-13b-base-v1.factprobe-replication-exposure-knows-trajectory-v1
factprobe-replication-exposure-knows-trajectory-v1
Does EXACT per-checkpoint cumulative corpus exposure predict whether OLMo-2 KNOWS a fact? Two measures per (checkpoint, relation): PAIRED (P(Yes|true) > P(Yes|hard-negative), the 'beats' defs) and MARGINAL (P(Yes|true), r2_p_true + per-feature Spearman). Checkpoints: after-phase1, base(s1+s2), SFT, DPO, RLVR + 7 pre-merge s2-anneal ingredients; both models; both relations (P26 spouse, P3373 sibling); surface/name level; all 7… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-exposure-knows-trajectory-v1.factprobe-replication-hardneg-knows-labels-surface-v1
factprobe-replication-hardneg-knows-labels-surface-v1
NAME-level (surface-form) paired 'knows the fact' judgements of OLMo-2 against hard negatives, across the training ladder. One row per subject NAME form (not the entity aggregate). subject_name held fixed; object side is the any-of max. Forward direction; spouse (P26) and sibling (P3373).
Dataset Info
Rows: 1299300
Columns: 11
Columns
Column
Type
Description
tag
Value('string')
7b or… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-hardneg-knows-labels-surface-v1.factprobe-replication-hardneg-knows-labels-v1
factprobe-replication-hardneg-knows-labels-v1
Paired 'knows the fact' judgements of OLMo-2 (7B, 13B) against GOOD hard negatives, across the training ladder. One row per (model, stage, relation, phrasing, true pair): P(Yes) on the true pair, best P(Yes) on the hardest hard negative, and beats_all (does the model prefer the true partner). Forward direction; spouse (P26) and sibling (P3373).
Dataset Info
Rows: 284260
Columns: 10
Columns
Column… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-hardneg-knows-labels-v1.factprobe-replication-b2-olmo7b-stage1-v1
factprobe-replication-b2-olmo7b-stage1-v1
Arm b2 of the FactProbe replication (He et al. arXiv:2503.22362): few-shot completion probing of the BASE OLMo-2-1124-7B final stage-1 checkpoint on symmetric-relation fact recognition (spouse P26, sibling P3373), forward and backward directions, all alias variations. One row per prompt; p_yes/p_no are next-token probability mass over Yes/No (question) or True/False (statement) single-token variants.
Dataset Info
Rows:… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-b2-olmo7b-stage1-v1.factprobe-replication-SUPERSEDED-stage1-counts-olmotok-v1
SUPERSEDED - do not use
Renamed 2026-08-25. Use latkes/factprobe-replication-stage1-counts-canonical-v1 instead.
It was counted with a name list missing each entity's canonical Wikidata name for 10,471 of 42,235 entities (24.8%). "Barack Obama" is not in it, and occurs 38,978,811 times in the corpus it claims to count.
It is kept only so earlier numbers can be traced to where they came from. Nothing current should read it.… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-SUPERSEDED-stage1-counts-olmotok-v1.factprobe-replication-stage1-fact-counts-v1
Correction, 2026-08-26
An earlier version of this card said the corpus states facts asymmetrically
in a way that tracks entity frequency, and gave 70.9% as the figure. That
number is right, and the generalisation drawn from it was wrong.
It holds for spouse and for no other relation. Recomputed across all four:
relation
pairs with a fact sentence
written more often with the MORE frequent entity first
spouse
1,265
70.9%
sibling
203
36.5% — the opposite
twinned town… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stage1-fact-counts-v1.factprobe-replication-stage1-counts-canonical-v1
factprobe-replication-stage1-counts-canonical-v1
Occurrences of every probed entity name in the OLMo-2 PRETRAINING corpus (olmo-mix-1124, 1,117 token files, 14.10 TiB), counted as OLMo token sequences with a word boundary required at each end, both written forms kept separate. SUPERSEDES factprobe-replication-stage1-counts-olmotok-v1, which was missing each entity's canonical name: the released triples name entities by their Wikidata aliases, a field that by construction… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stage1-counts-canonical-v1.factprobe-replication-b2-olmo13b-stage1-v1
factprobe-replication-b2-olmo13b-stage1-v1
Arm b2 of the FactProbe replication (He et al. arXiv:2503.22362): few-shot completion probing of the BASE OLMo-2-1124-7B final stage-1 checkpoint on symmetric-relation fact recognition (spouse P26, sibling P3373), forward and backward directions, all alias variations. One row per prompt; p_yes/p_no are next-token probability mass over Yes/No (question) or True/False (statement) single-token variants.
Dataset Info
Rows:… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-b2-olmo13b-stage1-v1.factprobe-replication-SUPERSEDED-stage2-counts-olmotok-v1
SUPERSEDED - do not use
Renamed 2026-08-25. Use latkes/factprobe-replication-stage2-counts-canonical-v1 instead.
It was counted with a name list missing each entity's canonical Wikidata name for 10,471 of 42,235 entities (24.8%).
It is kept only so earlier numbers can be traced to where they came from. Nothing current should read it.
factprobe-replication-stage2-counts-olmotok-v1
Occurrences of every probed entity name in the OLMo-2-7B mid-training corpus (the… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-SUPERSEDED-stage2-counts-olmotok-v1.factprobe-replication-excluded-names-v1
factprobe-replication-excluded-names-v1
The complete list of probed names whose occurrence counts measure an ordinary English word rather than the entity. 1,154 names: 119 that came with He et al.'s released triples and 1035 that were added on 2026-08-24 when a quarter of the entities got their canonical Wikidata name back. Q254046 is a commune in the Ardennes called 'This'; Q1022407 and Q1809719 are both called 'Police'. Wikidata also lists song and film titles as alternative… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-excluded-names-v1.factprobe-replication-stagematched-7b-v1
Stage-matched probing of OLMo-2-7B (49,218,520 rows)
Five training stages (S1 end of pretraining, S2 released base, S3 SFT, S4 DPO,
S5 Instruct) x four Wikidata relations (P26 spouse, P3373 sibling, P190
twinnedTown, P47 bordersWith) x two phrasings (question, statement), each run
with 1:1 scrambled negatives drawn with a fixed seed so the negative set is
identical at every stage. Few-shot prompt held fixed across all stages; only
the model weights differ.
column
meaning… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stagematched-7b-v1.factprobe-replication-stage2-counts-canonical-v1
factprobe-replication-stage2-counts-canonical-v1
Occurrences of every probed entity name in the OLMo-2-7B mid-training corpus (576 token files), counted as OLMo token sequences with a word boundary required at each end, both written forms kept separate. SUPERSEDES factprobe-replication-stage2-counts-olmotok-v1, which was missing each entity's canonical name: the released triples name entities by their Wikidata aliases, and Wikidata keeps the canonical name in a separate field. 9… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stage2-counts-canonical-v1.factprobe-replication-stage2-13b-counts-v1
factprobe-replication-stage2-13b-counts-v1
Occurrences of every probed entity name in the OLMo-2-13B mid-training corpus: 1,102 token files, 1.1 TiB, 3,871,962,845 occurrences over 137,128 names of the 174,729 searched. Counted as OLMo token sequences with a word boundary required at each end, both written forms kept separate, and including each entity's canonical Wikidata name -- which a quarter of entities were missing. This is the last piece that was needed for the 13B to be… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stage2-13b-counts-v1.factprobe-replication-canonical-names-added-v1
factprobe-replication-canonical-names-added-v1
The canonical name that was missing for 10,471 of the 42,235 probed entities (24.8%), and had therefore never been counted in the corpus nor put to the model. He et al.'s released triples name entities by their Wikidata aliases -- the 'also known as' field -- which by construction excludes the canonical name. Q76 carried 31 names, none of them 'Barack Obama'. Each row also records whether the added name is an ordinary English word… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-canonical-names-added-v1.factprobe-replication-stage2-fact-counts-v1
factprobe-replication-stage2-fact-counts-v1
How often the 7B mid-training corpus states each fact in so many words -- 'Netherlands borders Germany' -- rather than merely naming both entities in one document. 383,132 sentences were searched: four phrasings per relation, both directions, each entity written with its canonical name. 1,305 occur at all, 6,660 occurrences in total. This corpus is 0.19 TiB; the same measurement over the 14 TiB pretraining corpus is being counted now… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-stage2-fact-counts-v1.