philipjohnbasile/hy3-demolition-mlx-reap25-v1

Hy3-Demolition-MLX reap25-v1

Explore the model guide · All public work

Release at a glance

A standalone, fused MLX model directory built from Tencent Hy3 (295B MoE, 21B active, 192 experts / top-8, 80 layers), rare-expert-preserving REAP-pruned to 144 experts per layer (25%) and then healed with a LoRA on verifier-filtered agent data. It is the same recipe as lite-v1 plus the prune — a smaller, lighter daily driver (223B params, ~87 GB peak vs lite-v1's 295B / 112 GB) with close results on the project suite and a documented tool-call format regression.

This is a research artifact, shared as-is with receipts: no support or fitness claims, Apache-2.0 like its base model. Recorded measurements and experimental projections are distinguished below; receipts live in the source repo.

Why reap25 (and why not deeper)

REAP (arXiv:2510.13999) ranks experts by the mean over routed tokens of gate_value × ‖expert_output‖₂ — a criterion that decouples frequency from impact, so rare high-impact experts are protected. The full family was built and measured on an M5 Max 128 GB:

reap25's single eval miss was a truncation of an otherwise-correct answer, not a wrong answer; it held every hard and brutal code/tool/JSON case (8/8, 8/8). reap40 scored the same count but failed a brutal code case with a runtime crash — so 25% is the pruning knee; deeper is not shipped.

Usage

⚠️ Runtime prerequisite — read first. This is a hy_v3 (Tencent Hy3) model. hy_v3 is not in mainline `mlx-lm` yet (ml-explore/mlx-lm#1211), so stock `mlx-lm`, LM Studio, and Ollama cannot load it (you'll get ValueError: Model type hy_v3 not supported). Run it with the pinned fork: ``bash pip install "mlx-lm @ git+https://github.com/eauchs/mlx-lm@a7cc3054b1ff48c19950513b422a66cfed7baa60" ` Reapply the FP32-router patch documented in [RESTORE.md](https://github.com/PhilipJohnBasile/hy3-demolition-mlx/blob/main/RESTORE.md). The pinned environment is the recorded release setup, not a fresh runtime qualification. A future upstream merge or release still needs loader and generation validation. Upstream PR #1211 remained open when checked on September 10, 2026. — For a model that runs on **stock mlx-lm / LM Studio today**, use the fast sibling [hy3-family-mini-qwen35b-v1`](https://huggingface.co/philipjohnbasile/hy3-family-mini-qwen35b-v1).

MLX only. Requires an Apple Silicon Mac with enough unified memory (measured peak 86.7 GB — leaves ~41 GB free on a 128 GB machine).

mlx_lm.generate --model <this-directory> --prompt "Write a tiny Python fizzbuzz." --max-tokens 512
mlx_lm.chat --model <this-directory>
mlx_lm.server --model <this-directory> --port 8080   # OpenAI-compatible /v1

No adapter path, agent framework, or custom runtime wrapper is required — the directory is the whole artifact. For clean, direct answers, serve with --chat-template-args '{"reasoning_effort":"no_think"}'; use high for hard agent tasks where step-by-step reasoning helps (the model reasons inside <think:opensource>…</think:opensource> tags). Suggested settings for coding/agent work: temperature 0–0.2, top-p 1.0.

⚠️ *`max_tokens` too low fails silently, not loudly. With reasoning left on (the default), a small token budget can be entirely consumed by the `<think:opensource>` trace, leaving zero output with no error* — this looks like a clean empty result, not a truncation. Confirmed on the sibling model in a real production pipeline (a max_tokens: 1600 cap returned zero results for every input before the cause was found; 7000–9000 fixed it). If output looks suspiciously empty, raise max_tokens or set reasoning_effort: no_think before assuming the task legitimately had nothing to produce.

⚠️ For strict-JSON/structured output, pin `repeat_penalty: 1.0` explicitly in the request — don't trust a server's UI/global default. A nonzero repeat penalty punishes exactly the repeated structural tokens JSON needs ("name":, "type":, array delimiters), and can silently corrupt output on batch/array-heavy extraction tasks. Confirmed in production: a serving UI had repeat_penalty: 1.1 active globally, invisible to the request itself, on a strict-JSON extraction workload. A pipeline's full sampling contract (temperature, top-k, repeat-penalty) belongs in every request body, never assumed from server state — server defaults and UI settings can change between calls or restarts without you noticing.

Serving multiple concurrent requests: mlx_lm.server (above) does batch requests, but was measured unreliable at 8-way concurrency (dropped ~25% of connections under load in testing). At ≤4 concurrent it's solid — 2.72× aggregate throughput, 4/4 requests completed. For reliable high-concurrency serving specifically, oMLX is a viable alternative: loads this checkpoint with one dependency swap (install this project's pinned mlx-lm fork into its venv instead of its default mainline pin — RESTORE.md has the exact command), and measured 3.49× aggregate at 8/8 concurrent requests with no drops, plus active memory-ceiling enforcement and a restart-surviving KV cache. Not a blanket "faster" claim — single-stream speed is comparable either way (~8.6 vs ~7.7-8.1 tok/s) — it's specifically the reliability and ceiling-safety at higher concurrency that's the win.

Experimental SSD streaming

The source project includes an expert pager that reads the same stored weights on demand, keeping a smaller cache resident. Its cache-budget experiments were measured on an M5 Max, not separately on physical 16/32/64 GB machines. The tier names below describe the intended budgets.

These are prototype observations, not device-wide fit or output-equivalence guarantees. Reading unchanged weights is separate from validating the paging implementation and generated outputs. The fully resident path above remains the recorded release recipe.

Streaming design and observations.

Source and recipe

• —Base: ox-ox/Hy3-295B-Instruct-w2q3exp-AProjQ8-SExpQ8-OutQ8-MTP-mlx (mixed-quant MLX checkpoint of Tencent Hy3), AR-only view (num_nextn_predict_layers=0).
• —REAP calibration: streamed, true-criterion, facet-bucketed over a domain-matched pack; all 11 domain facets present in the saliency.
• —Prune: 192 → 144 experts/layer (25%), rare-expert-preserving plan verified by the analyzer (ACCEPT — 87% average saliency mass kept, no layer below 70%, all facets protected in every layer). Pruned tensors keep their original quantization (no dequant→requant cycle).
• —Heal LoRA: rank 8 on 8 layers, 200 iterations, batch 1, lr 1e-5, adamw, max-seq 2048, trained against the is_training template view (EOS-safe), val loss 1.566 → 0.979.
• —Fused with a streamed lazy fuse (one shard at a time), stock Hy3 chat template.

Build scripts, receipts, and the full recipe: https://github.com/PhilipJohnBasile/hy3-demolition-mlx

Verification receipts

All measured on the fused artifact, 2026-07-08, receipts committed under eval/receipts/ in the source repo:

• —Full eval (suite + hard + brutal) 45/46; hard 8/8, brutal 8/8; the one miss is a token-cap truncation of a correct domain answer, not a wrong answer (hy3_reap25_eval.jsonl, hy3_reap25_vs_lite_compare.json).
• —Manual side-by-side vs lite-v1 on real code / planning / repair / domain prompts (production no_think mode): clean, correct, direct, functionally indistinguishable — one marginal edge each way (reap25_vs_lite_side_by_side.md).
• —Prune-plan analyzer verdict: ACCEPT (hy3_reap25_plan_report.json).
• —Pruned + fused smokes: correct output, clean EOS stop (hy3_reap25_pruned_smoke.json, hy3_reap25_fused_smoke.json).
• —Peak inference memory 86.7 GB.

Limitations

• —The model learned self-checking and repair habits; it cannot execute compilers, tests, shells, or tools. Verifying its outputs still needs a real harness.
• —Quantized, pruned MoE base: expect quantization artifacts, and a small capability cost from the 25% prune (visible as +0.26 healed val loss over lite-v1, though not as felt output-quality loss in the manual pass).
• —Trained/evaluated primarily in English on agent/code tasks.
• —~~Decode speed is unchanged from lite-v1 (top-8 × 21B active is the same); the win is ~25 GB of memory headroom, not throughput.~~ Correction (2026-07-22): measured faster. The expectation was reasonable — top-8 × 21B active is identical — but the paired receipt in the source repo (eval/receipts/hy3_reap25_vs_lite_compare.json) measures reap25 at 8.341 tok/s vs lite-v1's 6.245 (+33.6%) on the same 46-case harness. Fewer resident experts buys throughput too, not just the ~25 GB of headroom. The original line understated the model and is struck rather than deleted.
• —Tool-calling tag format regression (found 2026-07-08, isolated against lite-v1): when emitting a native <tool_call:opensource> block, this model unreliably omits the <tool_sep:opensource> tag between the function name and its arguments — confirmed across 2 independent tool schemas/prompts. arg_key/arg_value pairs still parse correctly, but strict-format parsers expecting <tool_call>NAME<tool_sep>... will mis-extract the function name. lite-v1 (same LoRA heal, no prune) does not show this — it appears specific to the 25% prune, not the heal, and wasn't caught by our eval suite (no test case exercised this exact tag boundary). Workaround: extract name via a looser regex, or fall back to arg_key/arg_value pairs plus a looked-up/known tool name until a fix ships.

Runtime support

Supported: mlx_lm.generate, mlx_lm.chat, mlx_lm.server. Not supported (today): GGUF, llama.cpp, vLLM, SGLang, CUDA serving.

• —LM Studio / Ollama: pending — needs the hy_v3 architecture in mainline mlx-lm (ml-explore/mlx-lm#1211). Today these load only on the pinned fork (eauchs/mlx-lm@a7cc3054b1ff48c19950513b422a66cfed7baa60); future mainline and LM Studio versions require separate validation.