Osye/mlp-surgery-restored-top15

mlp-surgery — restore top 15 (raw) on Qwen2.5-3B

Qwen2.5-3B-Instruct that was fine-tuned on perplexity-filtered OpenHermes 2.5 (which damaged its reasoning), then partially restored by copying back the top-15 most-damaged MLP layers from the base model. No retraining. Just weight surgery.

Method (short)

1. 1.Take the broken finetune (mlp-surgery-broken).
2. 2.Score MLP layer parameters via raw gradient-norm scoring on the broken model's 100 GSM8K errors.
3. 3.Copy the top-15 from base into the broken model. Save.

Eval

lm-eval, GSM8K flexible-extract 5-shot, ARC Challenge accnorm 0-shot, no chat template, batchsize 8, single seed (2026-05-07).

This model is the "Restore top 15" row.

Companion models + code

• —mlp-surgery-broken — input baseline
• —mlp-surgery-restored-top5
• —mlp-surgery-restored-top15
• —mlp-surgery-restored-top30
• —mlp-surgery-restored-specificity-top10
• —Code: https://github.com/Malum0x/mlp-surgery

Caveats

Single seed. Magnitudes are ~1pt. The "no chat template" eval style means absolute numbers are below what you'd see with chat template applied (~78% GSM8K), but relative comparisons across the same setup are meaningful.