Datasetpublic

latkes/factprobe-replication-arm-a-olmo2-13b-instruct-v1

factprobe-replication-arm-a-olmo2-13b-instruct-v1 Arm (a) sanity replication of He et al. arXiv:2503.22362: their code, their Zenodo triples, their prompts (chat template, 36 alias variations, greedy). One row per (pair, direction, alias-variation) with the full generated text. Bucketing/analysis is done downstream (their analyse_experiment.py); OLMo cells reproduce their Tables 2/4 to +/-0.006. Dataset Info Rows: 5175168 Columns: 10 Columns… See the full description on the dataset page: https://huggingface.co/datasets/latkes/factprobe-replication-arm-a-olmo2-13b-instruct-v1.

sourceHugging Facemitupdated 1mo agoView on Hugging Face

1likes21downloads

Dataset Card

factprobe-replication-arm-a-olmo2-13b-instruct-v1

Arm (a) sanity replication of He et al. arXiv:2503.22362: their code, their Zenodo triples, their prompts (chat template, 36 alias variations, greedy). One row per (pair, direction, alias-variation) with the full generated text. Bucketing/analysis is done downstream (their analyse_experiment.py); OLMo cells reproduce their Tables 2/4 to +/-0.006.

Dataset Info

—Rows: 5175168
—Columns: 10

Columns

Column	Type	Description
relation	Value('string')	P26 spouse / P3373 sibling / P190 twinnedTown / P47 bordersWith
model	Value('string')	No description provided
template_type	Value('string')	No description provided
subject	Value('string')	No description provided
object	Value('string')	No description provided
direction	Value('string')	forward = CSV (s,o) order; backward = reversed
variation_index	Value('int64')	alias-pair variation index (up to 36 per pair)
model_response	Value('string')	full generated text for this prompt (untruncated)
answer_em	Value('bool')	exact match of lowercased/stripped response to yes/true
answer_in	Value('bool')	substring-containment variant (computed, unused in their analysis)

Generation Parameters

json

{
  "script_name": "refs/FactProbe/probe.py (upstream, verbatim; TP/max_model_len passthrough only)",
  "model": "olmo2-13b-instruct",
  "description": "Arm (a) sanity replication of He et al. arXiv:2503.22362: their code, their Zenodo triples, their prompts (chat template, 36 alias variations, greedy). One row per (pair, direction, alias-variation) with the full generated text. Bucketing/analysis is done downstream (their analyse_experiment.py); OLMo cells reproduce their Tables 2/4 to +/-0.006.",
  "hyperparameters": {
    "temperature": 0.0,
    "logprobs": 10,
    "batch_size": 500
  },
  "input_datasets": [
    "Zenodo 10.5281/zenodo.15092789"
  ],
  "experiment_name": "factprobe-replication",
  "job_id": "runpod:fp-13b",
  "cluster": "runpod-personal",
  "artifact_status": "final",
  "canary": false
}

Usage

python

from datasets import load_dataset

dataset = load_dataset("latkes/factprobe-replication-arm-a-olmo2-13b-instruct-v1", split="train")
print(f"Loaded {len(dataset)} rows")