CoolFace
Modelpublic

arianraje/mimo-7b-gdn-hybrid-2.0B-OPD

sourceHugging Facemitupdated 21d agoView on Hugging Face
0likes391downloads
Model Card

MiMo-7B GDN Hybrid — stage3-2.0B-OPD

Decayed final of the 2.0B-token rung of the MiMo on-policy-distillation (OPD) WSD ladder (1:4 uniform hybrid of Qwen2 attention and Gated DeltaNet layers, teacher XiaomiMiMo/MiMo-7B-RL-0530). Run mimo-mathmix-h32k-runpod-2p0b-v1-20260905; snapshots/final, step 12887, 2,000,126,188 consumed generation tokens; model window 65,536.

Recipe: extends arianraje/mimo-7b-gdn-hybrid-1.8B-OPD from its pre-decay state (1,721.4M tokens) with no recipe change — the same stage-3 math mixture, rollout horizon 32,768, sampler window 49,152, 600-step linear decay, queue factor 6.0. This is the ladder's first matched-recipe A/B at H=32K, so differences vs the 1.8B row are a pure tokens-scaling read (the 1.8B row's own confounds vs earlier rows carry forward unchanged). The pre-decay trunk of this rung is public at arianraje/mimo-7b-gdn-opd-predecay-1921m-step12391.

Evaluation (TABLES.md protocol, raw JSONs under full_eval/)

Metric1.8B-OPD**2.0B-OPD**
AIME24 think pass@1 / pass@8 @32K (n=30×8)63.3 / 76.759.6 / 80.0
AIME25 think pass@1 / pass@8 @32K (n=30×8)49.2 / 73.347.1 / 73.3
MATH-500 think pass@1 @32K93.493.8
MATH-500 no-think pass@1 @4K70.270.8
GSM8K no-think strict / flexible @1K58.5 / 65.258.1 / 62.2
MMLU 5-shot53.854.6
PIQA / HellaSwag / ARC-E / ARC-C / Winogrande72.0 / 60.3 / 62.8 / 39.6 / 59.772.5 / 59.9 / 62.7 / 39.5 / 58.8
NIAH multikey 4K / 8K / 16K / 32K (n=500)96.8 / 97.2 / 95.0 / 84.696.8 / 96.8 / 94.8 / 83.8
NIAH single, multiquery (all lengths)≥99.8≥100.0
trunc / mean gen tokens: MATH-500, AIME24, AIME253.6%/8168, 26.7%/21925, 32.5%/235523.8%/7930, 29.2%/21862, 38.3%/24357

AIME cells are n=30 problems (±~14 pts on pass@1) — trend data; MATH-500 (n=500) is the powered cell. full_eval/*.generations.json carry every sample.

Loading

Custom architecture; needs the mimo_gdn model registration from the training repo (transformers 4.57.x + flash-linear-attention 0.5.x), same as the other arianraje/mimo-7b-gdn-hybrid-*-OPD checkpoints.