ManniX-ITA/Ornith-1.5-27B-A3B-CoderX

Ornith-1.5-27B-A3B-CoderX

BF16 weights for CoderX — a code-targeted expert prune of Ornith-1.5-35B-A3B: 256 experts per layer reduced to 184, ~35.9B → ~26.7B, still A3B active. Router width, attention, norms and the MTP speculative head are untouched.

Same expert budget and the same competence map as its sibling `Ornith-1.5-27B-A3B-Coder` — the entire difference is which 184 experts survive.

Recipe

Our per-category competence map drives selection, with a REAP-style stability floor (`--protect 6`) layered on top: the 6 experts per layer that our ranking would drop but REAP ranks highest are rescued, paid for by evicting the 6 lowest-ranked experts among our keeps. Verified against the Coder map: exactly 6 experts per layer differ, on all 40 layers (0/40 layers identical).

The 72 evicted experts per layer are discarded, not merged. The arm is built with --merging none, so every retained expert is bit-identical to its counterpart in the base, and the build asserts it rather than assuming it (verify_arm_identity.py → ARM_IDENTITY_OK). No fine-tuning, no distillation, no expert folding.

Maps and recipe (reproducible, in the open)

Everything that produced this checkpoint lives in **omnimergekit**, under `recipes/ornith_1_5_35b_a3b_prune/`:

The three rungs are a genuine nested dose axis rooted at Coder — the set of experts each rung changes relative to Coder satisfies E6 ⊂ E12 ⊂ E24 on all 40 layers, verified from the maps themselves, so the ladder measures dose and not three unrelated cuts.

Quantised builds

GGUF (imatrix at every tier, MTP head included): **`Ornith-1.5-27B-A3B-CoderX-MTP-GGUF`**.

Evaluation

Q6K + imatrix, `llama.cpp`, **`sampler=recommended`** (temp 0.6 / topp 0.95 / top_k 20 — the vendor profile for this family; greedy is not viable here), identical templates and identical prompts across the three columns.

The headline is LiveCodeBench v6: 0.7662 against the 256e teacher's 0.6623, +10.4 pp while carrying 28% fewer experts, and +3.9 pp over the Coder sibling. CoderX also recovers most of the GPQA that Coder gives up (0.8131 vs 0.7677), which is the property the stability floor is supposed to buy: the rescued experts are the ones general reasoning needs.

Read the small gaps as ties. At n=77 one LCB problem is 1.30 pp and the unpaired binomial SE is ≈7.4 pp; at n=100 one problem is 1.0 pp. Every cell is a single sampled draw. The one repeat we hold is on Coder's LCB cell — two complete, independent 77-problem runs of the same configuration both scored 0.7273, with visibly different generation profiles (tok_p50 12,680 vs 12,654, 5 vs 4 cap-hits). Identical score, different draw. That is one repeat on one cell, not a variance band for the table.

Per language, MultiPL-E (CoderX / Coder / 256e): Rust 0.71 / 0.74 / 0.75 · Java 0.92 / 0.87 / 0.91 · JS 0.88 / 0.88 / 0.90. The CoderX↔Coder gap there is two problems out of 300 — a tie.

Mixed measurement basis, deliberately. The LCB cells for CoderX and Coder were measured on a patched llama.cpp in which a PEG parse failure degrades to content instead of throwing (which used to discard the generation entirely); the base cell predates that patch but recorded zero discarded rows, so it is unaffected. Nothing else differs — same quant, same server flags, same sampler, same prompts.

Tool-calling benchmark — tool-eval-bench hardmode (88 scenarios, 176 pts)

Ornith-1.5-27B-A3B-CoderX scores 134.0 ±2.5 of 176, ninth of ten — and this figure is a LOWER BOUND.

One of the five seeds (s42) is graded on 174 points, not 176: scenario TC-05 was dropped by an infrastructure fault on the serving side, not by a model failure. The cell was quarantined and re-run on 2026-09-09, and the fault reproduced, so the cell was kept and flagged rather than silently pooled. The plotted and tabulated value counts the dropped scenario as 0, which is why it is a floor. The excluded scenario sits in Parameter Precision, which is why that category reads 4/4 on that seed rather than 6/6.

Unlike its Coder sibling (136.8), the two are statistically indistinguishable here — overlapping CIs, and the ordering is inside the noise. Do not read CoderX as worse than Coder on this benchmark.

Category profile: perfect (100%) on Tool Selection, Restraint & Refusal, Toolset Scale and Creative Composition; Structured Reasoning 5.6/6 (93.3%). Weakest at Autonomous Planning 3.4/6 (56.7%), Context & State 12.6/20 (63.0%), Code Patterns 3.8/6 (63.3%) and Hard Mode 25.2/38 (66.3%).

14 safety-critical failures across five seeds: TC-60 on all five, TC-33 (Hallucination Resistance) and TC-58 (Fake System Message in File) on four each.

Note the quant: IQ4_XS, versus Q4KM for the A3B and Omnimerge rows.

[image]

Full cohort

* one seed (s42) is graded on 174 pts, not 176 — see that model's card.

<details> <summary><b>Basis — read before comparing these numbers to anything</b></summary>

• —Scorer: `tool-eval-bench` v2.6.0 (the pip/uv-installed package, verified via tool_eval_bench.__file__, not a git checkout). An earlier note in the runner claimed cf54b4b (v2.6.0-45); that is wrong and has been corrected — no cell ever ran it. All 50 cells ran the same v2.6.0, so the cohort is internally consistent.
• —v2.6.0 carries a known scorer crash on TC-62. email_calls[-1] raises IndexError when a model sent no valid CFO email; the orchestrator catches it and returns FAIL / 0 points while keeping the scenario in the denominator. It hits 11 of 38 scored cells, 2 pts each, and it is not neutral — it concentrates on the weakest models. Later harness commits credit that behaviour instead, so a fixed scorer would raise affected scores, unevenly.
• —5 paired seeds [42–46], 64k context, context-pressure 0.25, max 8 turns, 120 s timeout, thinking enabled, sampler temp 0.6 / top-p 0.95 / top-k 20 (not greedy).
• —Served on llama.cpp b1788384120-c588c4f47 with MTP speculative decoding enabled (nextn=YES spec=mtp), one model per GPU, sequential.
• —Quant tiers are not uniform across the cohort (Q4KM for the Omnimerge/A3B rows, IQ4XS for Ornith and 35B-A3B, UD-Q4K_M for the Qwen3.8 base). Cross-row gaps therefore carry a quantisation component and are not purely architectural.
• —Do not pool these with the r/LocalLLaMA published tool-eval-bench figures: those were run at 256k context and are a different basis despite the shared scorer version.

</details>

Cross-family reference — Qwen3.6-27B-A3B

The sibling program applied the same idea (256e → 184e, competence map, REAP stability floor) to Qwen3.6-35B-A3B. Those cells are sampler- and template-matched to the table above — every one records sampler=recommended, temp 0.6 / topp 0.95 / topk 20, Q6_K, llama.cpp — so they can share a table. Read from each run's summary.json, not from the sibling cards.

Only the five benches below ran the same template in both programs. The Qwen cohort used family-specific variants elsewhere (aime_30_qwen, gsm8k_100_boxed, math500_100_qwen, humaneval_full_think), and its cells on the plain templates this program used are not usable (Qwen CoderX scores 0.22 on math500_100 and the base 0.36 on gsm8k_100 — a template/parser mismatch, not capability). Those rows are omitted rather than cross-quoted.

What it says. The floor arm wins in both families — Qwen CoderX beats Qwen Coder by 14.3 pp on LCB, Ornith CoderX beats Ornith Coder by 3.9 pp — which is the strongest evidence that the stability floor is doing real work and not fitting one model's noise. Between families: Qwen CoderX is the better LiveCodeBench model (0.8312 vs 0.7662), Ornith CoderX the better MultiPL-E one (0.8367 vs 0.8000), and Ornith gives up a lot of instruction-following (IFEval 0.7600 vs 0.9300). These are different base models, so the column difference is not attributable to the prune.

Why these differ from the numbers on the Qwen cards. The Qwen program measured two cohorts. Its own cards quote the greedy one (lcb_v6_77q_48k, 24k think / 48k total, template_default T=0.0) — e.g. LCB CoderX 0.727. The table above uses the sampled qwen_suite cohort (lcb_v6_77q, recommended T=0.6), because that is the cohort whose template and sampler match this program's cells. Both cohorts are correct and each is labelled on its own card; they are simply not interchangeable.

Generation length and deliberation

Two things worth knowing before you budget context or wall time.

Where this family actually thinks. Nine of the eleven benches run in split reasoning regime and the model answers essentially directly: thinking_tokens_est is 0 on GPQA, AIME, MATH-500, GSM8K, IFEval, HumanEval and MultiPL-E, and rounds to 0.2–1.2% of completion on ARC. Only LiveCodeBench v6 runs inline, with a 12,288-token thinking budget inside a 32,768-token cap — and there the median generation is ~12.7k tokens, an order of magnitude above every other bench. This model ruminates only where the budget invites it.

Completion tokens, p50 / p90 / max, and cap-hits at the per-bench ceiling:

The one real regression is the LCB tail. Median length is indistinguishable across all three arms (12,690 / 12,680 / 12,697 tokens — within 0.1%), but both prunes run away on 5 of 77 problems, hitting the 32,768 ceiling where the base never exceeds 14,296 tokens. Those truncations score as failures, so CoderX's 59/77 is achieved despite them. This costs tail latency and is worth a stop condition in production; it does not affect the typical request.

Deliberation vs. answer, on LCB. There is no separable reasoning channel to report: these caches carry no `thinking` table, and llama.cpp's --reasoning-format deepseek strips the <think> delimiters — only 2–3 of 77 rows retain one. The split below therefore uses LCB's structural anchor: the submitted answer is the last fenced code block, everything before it is deliberation. It is a proxy for the reasoning/answer split, not a token count:

~94–95% of every LiveCodeBench generation is deliberation, and the code that actually gets graded is ~1.4k characters. All three arms deliberate at the same median; pruning did not make this model think more or less, it only widened the tail. If you want that tail bounded, cap it at serve time (--reasoning-budget) rather than hoping for a shorter draw.

Limitations

• —Single-draw evals. Only one cell in the table has a repeat, and it is on the Coder arm. Do not rank arms on a sub-2-problem margin.
• —The MultiPL-E row's cap asymmetry is described above; treat CoderX↔Coder there as a tie.
• —The 5-of-77 LCB runaway is a real tail-latency regression, inherited by both prunes.
• —Apache-2.0 · research checkpoint.