CoolFace
Datasetpublic

latkes/factprobe-replication-exposure-knows-trajectory-v1

factprobe-replication-exposure-knows-trajectory-v1 Does EXACT per-checkpoint cumulative corpus exposure predict whether OLMo-2 KNOWS a fact? Two measures per (checkpoint, relation): PAIRED (P(Yes|true) > P(Yes|hard-negative), the 'beats' defs) and MARGINAL (P(Yes|true), r2_p_true + per-feature Spearman). Checkpoints: after-phase1, base(s1+s2), SFT, DPO, RLVR + 7 pre-merge s2-anneal ingredients; both models; both relations (P26 spouse, P3373 sibling); surface/name level; all 7… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-exposure-knows-trajectory-v1.

sourceHugging Facemitupdated 25d agoView on Hugging Face
0likes45downloads
Dataset Card

factprobe-replication-exposure-knows-trajectory-v1

Does EXACT per-checkpoint cumulative corpus exposure predict whether OLMo-2 KNOWS a fact? Two measures per (checkpoint, relation): PAIRED (P(Yes|true) > P(Yes|hard-negative), the 'beats' defs) and MARGINAL (P(Yes|true), r2ptrue + per-feature Spearman). Checkpoints: after-phase1, base(s1+s2), SFT, DPO, RLVR + 7 pre-merge s2-anneal ingredients; both models; both relations (P26 spouse, P3373 sibling); surface/name level; all 7 exposure features. s2 soup variants kept separate; 7B DPO run both ways.

Dataset Info

  • —Rows: 54
  • —Columns: 63

Columns

ColumnTypeDescription
checkpointValue('string')cumulative checkpoint id (modelstage[variant/_dpoversion]); exposure = exactly the data it saw
probe_tagValue('string')No description provided
relationValue('string')P26 spouse or P3373 sibling
n_rowsValue('int64')surface (subject name, object name) true rows regressed
n_subjectsValue('int64')No description provided
aucbeats1Value('float64')No description provided
accbeats1Value('float64')No description provided
ratebeats1Value('float64')No description provided
aucbeats1f1subjValue('float64')No description provided
aucbeats1f2objValue('float64')No description provided
aucbeats1f3coocValue('float64')No description provided
aucbeats1f4subj_relValue('float64')No description provided
aucbeats1f5tripleValue('float64')No description provided
aucbeats1f6fact1Value('float64')No description provided
aucbeats1f7fact_allValue('float64')No description provided
aucbeats2Value('float64')No description provided
accbeats2Value('float64')No description provided
ratebeats2Value('float64')No description provided
aucbeats2f1subjValue('float64')No description provided
aucbeats2f2objValue('float64')No description provided
aucbeats2f3coocValue('float64')No description provided
aucbeats2f4subj_relValue('float64')No description provided
aucbeats2f5tripleValue('float64')No description provided
aucbeats2f6fact1Value('float64')No description provided
aucbeats2f7fact_allValue('float64')No description provided
aucbeats3Value('float64')No description provided
accbeats3Value('float64')No description provided
ratebeats3Value('float64')No description provided
aucbeats3f1subjValue('float64')No description provided
aucbeats3f2objValue('float64')No description provided
aucbeats3f3coocValue('float64')No description provided
aucbeats3f4subj_relValue('float64')No description provided
aucbeats3f5tripleValue('float64')No description provided
aucbeats3f6fact1Value('float64')No description provided
aucbeats3f7fact_allValue('float64')No description provided
aucbeatsallValue('float64')grouped held-out AUC predicting PAIRED knows = P(Yestrue) beats the BEST hard-negative
accbeatsallValue('float64')No description provided
ratebeatsallValue('float64')fraction of pairs where the model prefers the true partner over the best wrong one
aucbeatsallf1subjValue('float64')No description provided
aucbeatsallf2objValue('float64')No description provided
aucbeatsallf3coocValue('float64')No description provided
aucbeatsallf4subj_relValue('float64')No description provided
aucbeatsallf5tripleValue('float64')No description provided
aucbeatsallf6fact1Value('float64')No description provided
aucbeatsallf7fact_allValue('float64')No description provided
aucbeatsanyValue('float64')PAIRED AUC, beats the WORST hard-negative
accbeatsanyValue('float64')No description provided
ratebeatsanyValue('float64')No description provided
aucbeatsanyf1subjValue('float64')No description provided
aucbeatsanyf2objValue('float64')No description provided
aucbeatsanyf3coocValue('float64')No description provided
aucbeatsanyf4subj_relValue('float64')No description provided
aucbeatsanyf5tripleValue('float64')No description provided
aucbeatsanyf6fact1Value('float64')No description provided
aucbeatsanyf7fact_allValue('float64')No description provided
r2ptrueValue('float64')grouped held-out R^2 predicting the MARGINAL P(Yestrue) from the 7 features
spearmanptruef1subjValue('float64')No description provided
spearmanptruef2objValue('float64')No description provided
spearmanptruef3coocValue('float64')Spearman of name-pair co-occurrence with P(Yestrue) (the dominant feature)
spearmanptruef4subj_relValue('float64')No description provided
spearmanptruef5tripleValue('float64')No description provided
spearmanptruef6fact1Value('float64')No description provided
spearmanptruef7fact_allValue('float64')No description provided

Generation Parameters

json
{
  "script_name": "run_knows_analysis.py + analyze_namepair_7feat.py",
  "model": "allenai/OLMo-2-1124-7B and -13B (all training checkpoints + pre-merge ingredients)",
  "description": "Does EXACT per-checkpoint cumulative corpus exposure predict whether OLMo-2 KNOWS a fact? Two measures per (checkpoint, relation): PAIRED (P(Yes|true) > P(Yes|hard-negative), the 'beats' defs) and MARGINAL (P(Yes|true), r2_p_true + per-feature Spearman). Checkpoints: after-phase1, base(s1+s2), SFT, DPO, RLVR + 7 pre-merge s2-anneal ingredients; both models; both relations (P26 spouse, P3373 sibling); surface/name level; all 7 exposure features. s2 soup variants kept separate; 7B DPO run both ways.",
  "experiment_name": "factprobe-replication",
  "relations": "P26 (spouse), P3373 (sibling)",
  "measures": "paired beats_1/2/3/all/any (auc+rate); marginal r2_p_true + spearman_p_true_f1..f7",
  "caveats": "13B s1 counted x1.291 (interim uniform, exact order-replay deferred); fact features near-zero; near-negative pairs' exposure features NOT counted (deferred).",
  "artifact_status": "final",
  "canary": false,
  "hyperparameters": {},
  "input_datasets": []
}

Usage

python
from datasets import load_dataset

dataset = load_dataset("latkes/factprobe-replication-exposure-knows-trajectory-v1", split="train")
print(f"Loaded {len(dataset)} rows")