inside-out-replication
inside-out-replication-v2-probe-scores
inside-out-replication-v2-probe-scores
Internal (probe) scores: logistic-regression probe on hidden states, best layer chosen by dev K. Trained probe .pkl files and per-layer selection JSON attached to this repo.
Dataset Info
Rows: 1574024
Columns: 7
Columns
Column
Type
Description
question_id
Value('string')
Question identifier
answer
Value('string')
Answer string (full)
label
Value('string')
Judge label CORRECT/INCORRECT
probe_score… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-probe-scores.inside-out-replication-canary-v1
inside-out-replication-canary-v1
Canary run: 5 questions per relation, 50 samples, Llama-3-8B. Full pipeline E2E test.
Dataset Info
Rows: 482
Columns: 11
Columns
Column
Type
Description
relation
Value('string')
Wikidata relation (P26=spouse, P264=label, P176=manufacturer, P50=author)
question_id
Value('string')
Unique question identifier
question
Value('string')
Entity-centric question text
gold_answer
Value('string')
Ground truth answer from… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-canary-v1.inside-out-replication-results-v1
inside-out-replication-results-v1
Full Inside-Out replication: 3 models x 4 relations x 450 test questions x 1000 samples. Includes P(a|q), P_norm, P(True) V0/V1/V2, and probe scores.
Dataset Info
Rows: 1523595
Columns: 20
Columns
Column
Type
Description
question_id
Value('string')
Unique question identifier
answer
Value('string')
Model-generated answer
label
Value('string')
Judge verdict: CORRECT or INCORRECT
log_p_a_q
Value('float64')
Log… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-results-v1.inside-out-replication-v2-samples
inside-out-replication-v2-samples
Raw sampled answers: 1 greedy + 1000 temperature samples per question (temp=1.0 test/dev, temp=2.0 train). Full outputs, no truncation.
Dataset Info
Rows: 12000
Columns: 10
Columns
Column
Type
Description
question_id
Value('string')
Question identifier
relation
Value('string')
Wikidata relation
question
Value('string')
No description provided
gold_answer
Value('string')
No description provided
greedy… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-samples.inside-out-replication-v2-gemma-attn-ablation-large-v1
Inside-Out Replication V2 — Gemma 2x2 ablation at N=100 (confirmation)
Confirmation of today's N=25 negative result on google/gemma-2-9b-it / P26 / test, run at 4x larger N (100 question_ids → 35,612 candidate rows per
arm) to reduce subset noise. Same 2x2 design as the N=25 canary:
attn_implementation × query_pre_attn_scalar.
Headline result at N=100
| metric | arm0 (sdpa,256) | arm1 (eager,256) | arm2 (sdpa,224) | arm3 (eager,224) | max |Δ| |
|---|---|---|---|---|---|
|… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-gemma-attn-ablation-large-v1.inside-out-replication-v2-pamqfix-metrics-v1
Inside-Out Replication V2 — A.8.1 in-context P(a|q) re-score (FULL)
3 models × 4 relations × test split, all 12 cells. P(a|q)/P_norm rebuilt with
paper-faithful A.8.1 in-context tokenization (the answer is tokenized jointly
with the generation-prompt context, not naive-concatenated). P(True) and
verifier variants unchanged (their code path is unaffected). Probe is
DEFERRED in this phase — externals only.
Headline finding: the A.8.1 fix moves K by ≤ 0.001 everywhere… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-pamqfix-metrics-v1.