sleepyeldrazi/Ornith-1.5-REAP-20B-A3B-GGUF

Ornith-1.5-REAP-20B-A3B-GGUF

Experimental. An expert-pruned, recovery-fine-tuned, domain-calibrated-quantized slice of ornith-ai/Ornith-1.5-35B-A3B (MIT). 35.9B total → 20.3B total, ~3B active (256 → 132 routed experts per layer, 48.4% expert sparsity). Built with the REAP method, a light LoRA recovery pass, and quantizations calibrated on agentic-coding / ML-engineering data.

⚠️ This is a research artifact. Expert pruning removes knowledge. Performance was validated around agentic coding and ML engineering (Python/C++). General-domain behavior (translation, creative writing, broad knowledge, multilingual nuance, long-document prose) has not been tested and is expected to be worse than the 35B base.

Part 1 — What this is, and what it's for

Intended use

• —Local agentic coding / ML-engineering workflows (Python/C++, tool calling, repo work, debugging, quantization/model work) on ~16 GB VRAM GPUs and 24 GB unified-memory Macs (via llama.cpp).
• —The 20B checkpoint is a high-precision slice of the 35B MoE: same ~3B active path, same architecture (hybrid GDN linear-attention + softmax attention + MoE), 48.4% fewer experts to store — so the memory savings come from storage, not from a smaller active compute path.

Files

For IQ quantizations of the un-pruned 35B base, see AtomicChat/Ornith-1.5-35B-A3B-GGUF (the AD-IQ3S/IQ3XXS we measured against in Part 2 comes from there).

Where it should succeed

• —Tool calling and ML-engineering coding — the calibration/training domain. Expert pruning is near-lossless in-domain when calibrated on the target domain (REAP, [arXiv:2510.13999]); the recovery LoRA and the imatrix calibration were built from the same domain.
• —Quick, high-quality generations at 4–5 bits on the active 3B path — q5k_ud matches or exceeds the BF16 checkpoint's own perplexity on the held-out test text (see Part 2), i.e. the quantization costs less than the recovery pass gains back.

Where it may fail (be realistic)

• —General knowledge & language quality: the 124 pruned experts carried generic-domain knowledge. Expect degraded translation, creative writing, and open-ended knowledge recall vs the 35B base. Untested — treat as unknown.
• —Hard multi-step reasoning: long reasoning traces are the heaviest load on the remaining experts; expect more brittleness than the 35B on very long chains.
• —Low-bit variants (ud_iq4nl): the IQ3S/IQ4NL expert map costs ~0.26 PPL on general text — fine for experimentation, not the quality pick.
• —MTP: the multi-token-prediction head was stripped (not part of the reaped checkpoint).

Serving tip — loop avoidance

In multi-compaction, long-horizon, difficult ML-engineering tasks, two near-loop episodes have been observed (the model circling on "Let me reconsider" fallback tokens in its reasoning; Liquid-AI-style doom-loop mechanism #1). Both escaped on its own after ~10k tokens of looping — there is no evidence yet of the model being unable to escape by itself. If you see time-wasting loops in your own workload, these two flags are a very light nudge:

llama-server ... --repeat-penalty 1.01 --repeat-last-n 256
# or via the API sampling params: "repeat_penalty": 1.01, "repeat_last_n": 256

Keep repeat_last_n around 256 so legitimately repeated identifiers in code are not penalized; the effect on normal output is negligible.

A note on quality vs the 35B bases

In limited personal testing, on ML-engineering tasks, q5k_ud (14.3 GB) outperformed IQ3-level quantizations of Qwen3.6-35B and of Ornith-1.5-35B at roughly the same file size. Be clear-eyed about what that means: in full precision, the reaped 20B probably does not outperform the 35B bases — it is a domain-focused, high-precision slice of one of them, sized for a workflow that wouldn't otherwise fit.

Part 2 — Everything we know (pipeline, data, measurements)

Pipeline (executed + designed)

1. 1.Trace generation. Served the 35B (NVFP4 W4A16, ModelOpt PTQ) and generated on-policy traces from a domain mix. Corpus: 2,387 prompts → ~5.3M tokens, splits ~50% tool-calling / ML-engineering, ~20% agentic multi-turn, ~15% math, ~15% general; 99.4% of rows completed with a final answer (0.6% thinking-only). All rows are the model's own chat-formatted generations (reasoning + answer).
2. 2.REAP (Router-weighted Expert Activation Pruning) — [arXiv:2510.13999], Cerebras repo (ICLR 2026). Saliency S_j = mean(g_j · ||f_j||₂) over the calibration tokens; 256 → 132 experts/layer (48.4%), top-8 routing preserved. Calibration used the domain mix above (code/ML-eng 35%, tool 25%, agentic 20%, math+general 20%). Includes the router-weight renormalization fix from the Cerebras repo.
3. 3.Recovery LoRA. Unsloth 4-bit QLoRA on the reaped checkpoint (rank 16, ~2 epochs over the completed traces; targets: attention q/k/v/o, GDN linear-attention projections, shared expert) with a small anti-hallucination abstention subset (~11%) — "I don't know" responses for obscure technical queries — blended into the recovery mix.
4. 4.*Anti-doomloop (FTPO / Antidoom) — designed, data prepared, not applied to this release. Liquid AI's [Antidoom](https://github.com/Liquid4All/antidoom) / [blog](https://www.liquid.ai/blog/antidoom): sample low-temperature completions, detect repeated spans, train a LoRA with Final Token Preference Optimization on the loop-starting token vs plausible alternatives (logit-space KL, two-part regularisation). An 18k-prompt code/math/tool subset was prepared for this stage. It was not run for this release because the measured loop proxy on the evaluation battery was already low (n-gram self-overlap 0.026–0.029* across variants); the stage remains the recommended follow-up if looping is observed in practice. (Liquid's published results: LFM2.5-2.6B 10.2%→1.4% loop rate, Qwen3.5-4B 22.9%→1%.)
5. 5.Quantization. Converted to GGUF F16, then computed an importance matrix (imatrix) via llama.cpp on ~3M tokens of chat-formatted calibration text rendered from the generated traces (same domain mix) — instruct models need chat-formatted imatrix calibration, not raw text. Quantized variants with the imatrix, plus a UD-style per-tensor map (Unsloth Dynamic-3.0-style important-tensor raises: attention/ssm-out/shared-expert Q80, norms F32, deepest-layer expert downs at Q6K) lifted from Unsloth's published Qwen3.6-35B map (same architecture family; tensor names transfer 1:1).

Related work referenced: LFM2 Technical Report (arXiv:2511.23404) — the tempered decoupled Top-K teacher loss (used conceptually for the recovery/distillation thinking), Liquid's Quantization-Aware Distillation (blog), NVIDIA ModelOpt NVFP4 (the serving path used for trace generation), and Unsloth Dynamic 3.0 (docs).

Perplexity (KLD) on held-out text

Same 382 KB held-out text (chat-format mixed domains, not in any calibration corpus), 25 × 4096-token chunks, greedy, llama.cpp, Q8_0 35B as reference:

Readings:

• —The REAP cut costs +0.66 PPL on general text (~18% relative) — expected: 48.4% of experts removed, generic text exercises all experts. On the calibration domain (agentic coding / ML engineering) the cut is near-lossless, consistent with REAP's domain-calibration findings.
• —The recovery LoRA moved generic-text perplexity by ~0.000 (4.3411 both) — it tuned the domain, not the base distribution.
• —*The quantization improved the recovered BF16's PPL (4.2730 < 4.3411)* — the domain imatrix + important-tensor raises regularize the low-bit experts into a slightly better distribution on this text.
• —The 35B IQ3 (3.7603) still beats q5k_ud (4.2730) on general-text PPL — the full model wins on generality; the reaped model wins in the targeted domain at a comparable size.

Evaluation battery (recovered vs pre-recovery)

203 fixed prompts: tool 75 / code 45 / math 30 / abstention probes 30 / doom-prone 23 (temp 0.3, up to 8k tokens, 64-way):

Both are effectively indistinguishable on these coarse axes — the recovery LoRA neither regressed nor visibly improved generic behavior; its value is in-domain fluency.

Quant variant comparison (all from the recovered BF16, with the domain imatrix)

Technical notes & caveats

• —The checkpoint is text-only (vision wrapper stripped; no vision weights exist in the source). GDN linear-attention kernels in llama.cpp co-process on CPU on some devices — expect prefill to be heavier than pure-GPU dense models.
• —IQ-type (imatrix) quants cost ~5–10% decode speed vs non-imatrix at the same size.
• —The MTP head is stripped (converted with --no-mtp); no speculative decoding.
• —ssm_out (linear-attention state output) is kept at Q8_0 (per Unsloth's Qwen3.5-family guidance, quantizing it hurts).
• —The BF16 folder is the full reaped+recovered checkpoint (Qwen3_5MoeForCausalLM, 132 experts, text-only config) — use it to re-quantize with different maps/imatrixes.
• —The 35B IQ3 numbers in Part 2 were measured on AtomicChat's AD-IQ3_S/IQ3_XXS GGUF — the un-pruned base, included here as a reference data point only.
• —Untested: translation, creative writing, long-document prose, multilingual, medical/legal QA, and other general domains. Use accordingly.