Datasetpublic

latkes/inside-out-replication-v2-gemma-old-tokenizer-v1

Inside-Out Replication V2 — Gemma OLD-tokenizer canary Negative-result canary on google/gemma-2-9b-it / P26 / test (25-qid subset), testing whether the HF tokenizer-files update (commit 4efc01a1, "Update tokenizer files (#36)", 2024-08-07) is a candidate cause of our Gemma P(True) anomaly (~0.64 vs paper-implied ~0.47). Two arms — same model weights, same scoring code, same data, same attn implementation (sdpa) and query_pre_attn_scalar=256. Only the tokenizer revision differs:… See the full description on the dataset page: https://huggingface.co/datasets/latkes/inside-out-replication-v2-gemma-old-tokenizer-v1.

sourceHugging Faceupdated 4mo agoView on Hugging Face

0likes8downloads

Dataset Card

Inside-Out Replication V2 — Gemma OLD-tokenizer canary

Negative-result canary on google/gemma-2-9b-it / P26 / test (25-qid subset), testing whether the HF tokenizer-files update (commit 4efc01a1, "Update tokenizer files (#36)", 2024-08-07) is a candidate cause of our Gemma P(True) anomaly (~0.64 vs paper-implied ~0.47).

Two arms — same model weights, same scoring code, same data, same attn implementation (sdpa) and query_pre_attn_scalar=256. Only the tokenizer revision differs:

—new_tokenizer: 11c9b309 (current latest, Aug 27 2024)
—old_tokenizer: 1937c70277fc (Jul 10 2024, immediately before PR #36)

Headline result

metric	new (Aug)	old (Jul)	Δ
logpa_q	0.5462	0.5462	0.0000
logpnormaq	0.5510	0.5510	0.0000
p_true	0.6888	0.6888	0.0000
logpaqeot	0.5659	0.5659	0.0000
logpnormaq_eot	0.5622	0.5622	0.0000
verifv1truefalse	0.6974	0.6974	0.0000
verifv2yesno	0.6922	0.6922	0.0000

Δ = 0.0000 on every metric. The tokenizer update didn't change tokenization of the entity answers we score on P26 (likely a tokenizer.json fast-tokenizer addition or special-token metadata change that doesn't affect typical entity strings).

Tokenizer revision is ruled out as a candidate cause of the Gemma anomaly.