datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
inside-out-replication-v2-probe-scores
inside-out-replication-v2-probe-scores
Internal (probe) scores: logistic-regression probe on hidden states, best layer chosen by dev K. Trained probe .pkl files and per-layer selection JSON attached to this repo.
Dataset Info
Rows: 1574024
Columns: 7
Columns
Column
Type
Description
question_id
Value('string')
Question identifier
answer
Value('string')
Answer string (full)
label
Value('string')
Judge label CORRECT/INCORRECT
probe_score… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-probe-scores.inside-out-replication-canary-v1
inside-out-replication-canary-v1
Canary run: 5 questions per relation, 50 samples, Llama-3-8B. Full pipeline E2E test.
Dataset Info
Rows: 482
Columns: 11
Columns
Column
Type
Description
relation
Value('string')
Wikidata relation (P26=spouse, P264=label, P176=manufacturer, P50=author)
question_id
Value('string')
Unique question identifier
question
Value('string')
Entity-centric question text
gold_answer
Value('string')
Ground truth answer from… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-canary-v1.inside-out-replication-results-v1
inside-out-replication-results-v1
Full Inside-Out replication: 3 models x 4 relations x 450 test questions x 1000 samples. Includes P(a|q), P_norm, P(True) V0/V1/V2, and probe scores.
Dataset Info
Rows: 1523595
Columns: 20
Columns
Column
Type
Description
question_id
Value('string')
Unique question identifier
answer
Value('string')
Model-generated answer
label
Value('string')
Judge verdict: CORRECT or INCORRECT
log_p_a_q
Value('float64')
Log… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-results-v1.inside-out-replication-v2-samples
inside-out-replication-v2-samples
Raw sampled answers: 1 greedy + 1000 temperature samples per question (temp=1.0 test/dev, temp=2.0 train). Full outputs, no truncation.
Dataset Info
Rows: 12000
Columns: 10
Columns
Column
Type
Description
question_id
Value('string')
Question identifier
relation
Value('string')
Wikidata relation
question
Value('string')
No description provided
gold_answer
Value('string')
No description provided
greedy… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-samples.inside-out-replication-v2-gemma-attn-ablation-large-v1
Inside-Out Replication V2 — Gemma 2x2 ablation at N=100 (confirmation)
Confirmation of today's N=25 negative result on google/gemma-2-9b-it / P26 / test, run at 4x larger N (100 question_ids → 35,612 candidate rows per
arm) to reduce subset noise. Same 2x2 design as the N=25 canary:
attn_implementation × query_pre_attn_scalar.
Headline result at N=100
| metric | arm0 (sdpa,256) | arm1 (eager,256) | arm2 (sdpa,224) | arm3 (eager,224) | max |Δ| |
|---|---|---|---|---|---|
|… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-gemma-attn-ablation-large-v1.inside-out-replication-v2-pamqfix-metrics-v1
Inside-Out Replication V2 — A.8.1 in-context P(a|q) re-score (FULL)
3 models × 4 relations × test split, all 12 cells. P(a|q)/P_norm rebuilt with
paper-faithful A.8.1 in-context tokenization (the answer is tokenized jointly
with the generation-prompt context, not naive-concatenated). P(True) and
verifier variants unchanged (their code path is unaffected). Probe is
DEFERRED in this phase — externals only.
Headline finding: the A.8.1 fix moves K by ≤ 0.001 everywhere… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-pamqfix-metrics-v1.inside-out-replication-v2-gemma-old-tokenizer-v1
Inside-Out Replication V2 — Gemma OLD-tokenizer canary
Negative-result canary on google/gemma-2-9b-it / P26 / test (25-qid
subset), testing whether the HF tokenizer-files update (commit 4efc01a1,
"Update tokenizer files (#36)", 2024-08-07) is a candidate cause of our
Gemma P(True) anomaly (~0.64 vs paper-implied ~0.47).
Two arms — same model weights, same scoring code, same data, same attn
implementation (sdpa) and query_pre_attn_scalar=256. Only the tokenizer
revision differs:… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-gemma-old-tokenizer-v1.inside-out-replication-v2-pamqfix-scores-full
Inside-Out Replication V2 — full corrected raw scores (all 12 cells)
This is the dataset to recompute K and K* for P(True), P(a|q),
P_norm(a|q) (and the sensitivity variants) across all 3 models × 4
relations (12 "cells"), test split.
One row per unique (question, answer) candidate — the greedy answer + the
1,000 temperature-1 samples are deduplicated to unique strings, then scored
once each. ~1.57M rows. Pipeline: judge-bug-fixed labels (pure unique-verdict
"Scheme A") + A.8.1… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-pamqfix-scores-full.inside-out-replication-v2-metrics
inside-out-replication-v2-metrics
Headline K and K* metrics for Inside-Out replication V2 (Gekhman et al. 2025). One row per (model, relation, scoring_method) on the test split. K = mean fraction of correct>incorrect answer pairs ranked correctly; K* = fraction of questions with perfect ranking (K=1).
Dataset Info
Rows: 72
Columns: 9
Columns
Column
Type
Description
model
Value('string')
Subject model: llama3-8b, mistral-7b, or gemma2-9b
relation… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-metrics.inside-out-replication-v2-judge-labels
inside-out-replication-v2-judge-labels
Judge labels for deduplicated sampled answers. Method is exact_match where the normalized answer equals gold, else a Qwen2.5-14B per-relation CoT judge producing grade A/B/C/D.
Dataset Info
Rows: 2901127
Columns: 9
Columns
Column
Type
Description
question_id
Value('string')
Question identifier
answer
Value('string')
Deduplicated answer string (full, never truncated)
count
Value('int64')
How many of the… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-judge-labels.inside-out-replication-v2-gemma-attn-ablation-v1
Inside-Out Replication V2 — Gemma 2x2 ablation (attn_impl × query_pre_attn_scalar)
Negative result canary on google/gemma-2-9b-it / P26 / test (25-qid subset,
7,398 candidate rows per arm), testing both real candidate causes of our
Gemma P(True) anomaly (~0.64–0.69 vs paper-implied ~0.47):
arm
attn_implementation
query_pre_attn_scalar
hypothesis if this matches paper
arm0_sdpa_256
sdpa
256
baseline = our current setup
arm1_eager_256
eager
256
paper used eager (kept… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-gemma-attn-ablation-v1.inside-out-replication-v2-corrected-metrics-v1
Inside-Out Replication V2 — CORRECTED metrics (Option C)
K/K* for 3 models x 4 relations x scoring methods, from the corrected
pipeline: judge-postprocessing bug fixed (pure unique-verdict Scheme A),
paper-faithful Option-C train (greedy-correct + >=1 judged-incorrect),
corrected labels (flips match the no-GPU baseline exactly), paper-consistent
best layers (llama L11 / mistral L12 / gemma L24). Full test sets n~370-449.
Probe vs best-external K-gap (avg over 4… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-corrected-metrics-v1.inside-out-replication-v2-pamqfix-canary-v1
Inside-Out Replication V2 — P(a|q) A.8.1-fix CANARY
Corrected externals (04_external_scores.py with the A.8.1 in-context
tokenization fix) over a deterministic 25-question-id subset of
Mistral-7B-Instruct-v0.3 / P264 / test (5832 candidate (q,a) rows).
Bounded subset used to validate the fix end-to-end on MLL before the full
3×4 re-score (per preflight rev2 spec; see red_team_brief.md).
Canary self-validation
C2 rows scored: 5832 (no build_qa_sequence ValueError;… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-pamqfix-canary-v1.inside-out-replication-v2-external-scores
inside-out-replication-v2-external-scores
Per (question, answer) external scores with judge labels for Inside-Out V2. Covers P(a|q), P_norm(a|q), P(True) and two verification-prompt variants. Used to compute external K/K*.
Dataset Info
Rows: 1752198
Columns: 20
Columns
Column
Type
Description
question_id
Value('string')
Question identifier, e.g. P26_test_0000
answer
Value('string')
Full sampled answer text (never truncated)
label… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-external-scores.