interpretable-finetuning/qwen2.5-7b-unaliased

Qwen2.5-7B, un-aliased ChatML turn markers

This is NOT the published checkpoint. Qwen/Qwen2.5-7B with the ChatML turn markers <|im_start|> (151644) and <|im_end|> (151645) replaced by the corresponding rows from Qwen/Qwen2.5-7B-Instruct. Everything else is bit-identical to the original.

337 of 339 tensors byte-identical to Qwen/Qwen2.5-7B
  differs: lm_head.weight, rows [151644, 151645]
  differs: model.embed_tokens.weight, rows [151644, 151645]

Tokenizer, vocabulary, token ids, config and architecture are unchanged. The -Instruct rows are used because it shares this tokenizer and vocabulary, so the ids line up exactly.

Why

Qwen/Qwen2.5-7B cannot reliably emit <|im_end|>. Unlike Qwen2.5-1.5B — whose defect was 267 bit-identical embedding rows — here the rows are bitwise unique but functionally indistinguishable: <|im_end|>'s lm_head row sits in a cluster of 8078 rows at mean-centred cosine >= 0.99 (nearest neighbour 1.000). Logits are h . W_j, so rows pointing the same way compete for the same probability mass however their bits differ.

tie_word_embeddings: False — so lm_head is a separate matrix and its rows had to be written explicitly. A repair that only touched embed_tokens would not have changed emission at all.

A bitwise alias check passes this checkpoint. That is how it went unnoticed.

Measured effect

Organisms fine-tuned on the unpatched base essentially never end a turn: their clean-mode completions hit the generation cap in 98.4-100% of cases (a comparable model on a repaired base: ~20%). Swapping only the lm_head <|im_end|> row, with the same trained adapter:

Fisher exact p < 1e-5. The deficit it causes is easy to misread: a model that cannot stop produces truncated, rambling answers that score ~1 point below terminated ones on a capability judge, which looks like a weak fine-tune rather than a broken stop token.

Caveats

• —The fix makes the token reachable, not reliable. After the swap p(<|im_end|>) is 0.13-0.35 at rank 1; a model trained against a repaired row reaches ~0.99. Training consolidates it.
• —Static geometry does not establish emittability. Three tests failed to predict it: bitwise equality (passed this checkpoint), a norm floor (rejected a base that works), and projection dominance (ranks Qwen2.5-7B-Instruct, which works, below Qwen2.5-32B, which does not). Verify behaviourally — generate turns and count how many end.
• —Only the two ChatML markers were touched. Other special tokens were not examined.
• —The input embed_tokens rows are near-dead in the -Instruct checkpoint too, so copying them changes little; the lm_head row is what matters behaviourally.

Provenance

Created by scripts/qwen15_make_unaliased_base.py from Qwen/Qwen2.5-7B-Instruct. The repository carries MODIFIED_BASE_MODEL.json and .md recording the same facts, so a downloader who never reads this card still finds out.