CoolFace
Datasetpublic

latkes/factprobe-replication-b2-olmo7b-stage1-v1

factprobe-replication-b2-olmo7b-stage1-v1 Arm b2 of the FactProbe replication (He et al. arXiv:2503.22362): few-shot completion probing of the BASE OLMo-2-1124-7B final stage-1 checkpoint on symmetric-relation fact recognition (spouse P26, sibling P3373), forward and backward directions, all alias variations. One row per prompt; p_yes/p_no are next-token probability mass over Yes/No (question) or True/False (statement) single-token variants. Dataset Info Rows:… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-b2-olmo7b-stage1-v1.

sourceHugging Facemitupdated 2mo agoView on Hugging Face
0likes33downloads
Dataset Card

factprobe-replication-b2-olmo7b-stage1-v1

Arm b2 of the FactProbe replication (He et al. arXiv:2503.22362): few-shot completion probing of the BASE OLMo-2-1124-7B final stage-1 checkpoint on symmetric-relation fact recognition (spouse P26, sibling P3373), forward and backward directions, all alias variations. One row per prompt; pyes/pno are next-token probability mass over Yes/No (question) or True/False (statement) single-token variants.

Dataset Info

  • —Rows: 1012136
  • —Columns: 15

Columns

ColumnTypeDescription
relationValue('string')Wikidata relation (P26 spouse / P3373 sibling)
row_typeValue('string')'true' (real pair) or 'scrambled_control'
subjectValue('string')No description provided
objectValue('string')No description provided
directionValue('string')forward = CSV (s,o) order; backward = reversed surface order
template_typeValue('string')No description provided
alias_sValue('string')subject alias used in this prompt variation
alias_oValue('string')object alias used in this prompt variation
demoorderflippedValue('bool')No description provided
prompt_hashValue('string')No description provided
p_yesValue('float64')next-token probability mass on Yes/True single-token variants
p_noValue('float64')next-token probability mass on No/False single-token variants
yesnomassValue('float64')pyes + pno (reliability screen; arm gate median >= 0.5)
says_yesValue('bool')pyes > pno for this prompt
model_responseValue('string')argmax next token text (full, untruncated)

Generation Parameters

json
{
  "script_name": "score_b2.py",
  "model": "allenai/OLMo-2-1124-7B @ stage1-step928646-tokens3896B (base, final stage-1)",
  "description": "Arm b2 of the FactProbe replication (He et al. arXiv:2503.22362): few-shot completion probing of the BASE OLMo-2-1124-7B final stage-1 checkpoint on symmetric-relation fact recognition (spouse P26, sibling P3373), forward and backward directions, all alias variations. One row per prompt; p_yes/p_no are next-token probability mass over Yes/No (question) or True/False (statement) single-token variants.",
  "hyperparameters": {
    "temperature": 0.0,
    "max_tokens": 1,
    "logprobs_topk": 20,
    "demos": "4 fictional-pair demonstrations, deterministic order flip"
  },
  "input_datasets": [
    "Zenodo 10.5281/zenodo.15092789 (FactProbe triples)"
  ],
  "experiment_name": "factprobe-replication",
  "job_id": "mll:65541",
  "cluster": "mll",
  "artifact_status": "final",
  "canary": false
}

Usage

python
from datasets import load_dataset

dataset = load_dataset("latkes/factprobe-replication-b2-olmo7b-stage1-v1", split="train")
print(f"Loaded {len(dataset)} rows")