CoolFace
Datasetpublic

latkes/inside-out-replication-v2-corrected-metrics-v1

Inside-Out Replication V2 — CORRECTED metrics (Option C) K/K* for 3 models x 4 relations x scoring methods, from the corrected pipeline: judge-postprocessing bug fixed (pure unique-verdict Scheme A), paper-faithful Option-C train (greedy-correct + >=1 judged-incorrect), corrected labels (flips match the no-GPU baseline exactly), paper-consistent best layers (llama L11 / mistral L12 / gemma L24). Full test sets n~370-449. Probe vs best-external K-gap (avg over 4… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-corrected-metrics-v1.

sourceHugging Faceupdated 4mo agoView on Hugging Face
0likes5downloads
Dataset Card

Inside-Out Replication V2 — CORRECTED metrics (Option C)

K/K for 3 models x 4 relations x scoring methods, from the corrected* pipeline: judge-postprocessing bug fixed (pure unique-verdict Scheme A), paper-faithful Option-C train (greedy-correct + >=1 judged-incorrect), corrected labels (flips match the no-GPU baseline exactly), paper-consistent best layers (llama L11 / mistral L12 / gemma L24). Full test sets n~370-449.

Probe vs best-external K-gap (avg over 4 relations)

modelcorrectedcorrupted runpaper
llama3-8b+2.5%~+6%+14%
mistral-7b+31.6%~+9%+48%
gemma2-9b+5.9%~+6%+47%

Honest conclusion

  1. 1.The self-introduced judge bug was a real, large confound: Mistral's gap rose ~+9% (corrupted) -> +31.6% (corrected) — ~2/3 of Mistral's hidden-knowledge signal was suppressed by the bug. Core hypothesis (internal >> external) holds strongly for Mistral.
  2. 2.Partial replication: llama (+2.5%) and gemma (+5.9%) remain far below the paper (+14%, +47%); residual discrepancy unexplained by the judge bug (fixed) or train labeling (paper-faithful). Candidate causes: hidden-state token position, probe specifics, per-relation train size.

See notes/experiments/inside-out-replication-v2/investigation_report.md.