ManniX-ITA/Qwen3.5-4B-M3-Fisher

Qwen3.5-4B-M3-Fisher

OMv2 recipe (OBIM-lite + DAREx-q + EMR election) with diagonal Fisher information weighting as the importance signal driving DAREx-q sparsification. Best HumanEval result of the study (57.93%, +5.49 pp over recipe alone).

Note: An earlier version of this checkpoint silently dropped the Fisher signal due to a tensor-name prefix mismatch (Fisher files keyed by model.X from named_parameters(), but Qwen3.55's safetensors index uses `model.languagemodel.X`). This release uses a prefix-aware Fisher lookup; Fisher signal is now actually applied. The buggy variant was bit-identical to M2 — see comparison table below for the correction.

Sources

Weights: 0.55 (A) / 0.45 (B). Density: 0.53. Seed: 42.

Fisher: 64×256-token fp32 calibration each source (see fisher/ subdir for the actual safetensors and methodology).

Phase 1+2 comparison (Q6_K)

Eval methodology: llama-server (--reasoning-format deepseek --reasoning-budget 8192 --parallel 2 --cache-type-k q8_0 --cache-type-v q8_0 -c 32768) → lm_eval local-completions against raw /v1/completions, temperature 0, maxgentoks=2048. All five variants scored under identical conditions. Floor reference: the top three rows are the base model and the two source fine-tunes evaluated under the same methodology. Hard finding: no merge beats `Qwen3.5-4B` base on HumanEval (best merge M3 = 57.93%, base = 60.37%). On MBPP, M4-v2 (52.20%) and M5 (51.40%) clearly beat both sources (best source: Crow at 48.20%) — merging does add MBPP capability, but borrows HumanEval ground.

Other variants in this study

• —Qwen3.5-4B-M1-Dare-Ties — vanilla DARE-TIES
• —Qwen3.5-4B-M2-OMv2 — OMv2 recipe (no importance signal)
• —Qwen3.5-4B-M3-Fisher — OMv2 + Fisher
• —Qwen3.5-4B-M4-ex-LRP — mergekit PR #682 ex-LRP
• —Qwen3.5-4B-M4-v2-ex-LRP-turbo — mergekit PR #682 turbo retest
• —Qwen3.5-4B-M5-OMv2-LRP — OMv2 + LRP