Datasetpublic

latkes/factprobe-replication-hardneg-knows-labels-surface-v1

factprobe-replication-hardneg-knows-labels-surface-v1 NAME-level (surface-form) paired 'knows the fact' judgements of OLMo-2 against hard negatives, across the training ladder. One row per subject NAME form (not the entity aggregate). subject_name held fixed; object side is the any-of max. Forward direction; spouse (P26) and sibling (P3373). Dataset Info Rows: 1299300 Columns: 11 Columns Column Type Description tag Value('string') 7b or… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-hardneg-knows-labels-surface-v1.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

0likes40downloads

Dataset Card

factprobe-replication-hardneg-knows-labels-surface-v1

NAME-level (surface-form) paired 'knows the fact' judgements of OLMo-2 against hard negatives, across the training ladder. One row per subject NAME form (not the entity aggregate). subject_name held fixed; object side is the any-of max. Forward direction; spouse (P26) and sibling (P3373).

Dataset Info

—Rows: 1299300
—Columns: 11

Columns

Column	Type	Description
tag	Value('string')	7b or 13b
stage	Value('string')	s1 end of stage-1 pretraining, s2 released base (post stage-2 anneal), s3 SFT, s4 DPO, s5 RLVR
relation	Value('string')	P26 spouse or P3373 sibling
template	Value('string')	statement or question
subject	Value('string')	subject entity QID
subject_name	Value('string')	the subject SURFACE FORM (the unit of analysis)
object	Value('string')	true-partner QID
p_true	Value('float64')	best P(Yes) for this name form over the partner's names
pnegmax	Value('float64')	best P(Yes) over the hardest hard negative
beats_all	Value('bool')	ptrue > pneg_max (the 'knows it' label)
margin	Value('float64')	ptrue - pneg_max

Generation Parameters

json

{
  "script_name": "dump_knows_labels_surface.py",
  "model": "allenai/OLMo-2-1124-7B and 13B, 5 training stages each",
  "description": "NAME-level (surface-form) paired 'knows the fact' judgements of OLMo-2 against hard negatives, across the training ladder. One row per subject NAME form (not the entity aggregate). subject_name held fixed; object side is the any-of max. Forward direction; spouse (P26) and sibling (P3373).",
  "experiment_name": "factprobe-replication",
  "cluster": "mll",
  "artifact_status": "final",
  "canary": false,
  "hyperparameters": {},
  "input_datasets": []
}

Experiment Documentation

For complete experiment details, see docs/results-hard-negatives.md

Usage

python

from datasets import load_dataset

dataset = load_dataset("latkes/factprobe-replication-hardneg-knows-labels-surface-v1", split="train")
print(f"Loaded {len(dataset)} rows")