Modelpublic

leonsarmiento/Orion-26B-A4B-v1-6bit-XL-mlx

sourceHugging Faceupdated 23d agoView on Hugging Face

0likes131downloads

Model Card

leonsarmiento/Orion-26B-A4B-v1-6bit-XL-mlx

This model was converted to MLX format from `TheDrummer/Orion-26B-A4B-v1` using BaseQuant_XL 6/8-bit mixed quantization optimized for Apple Silicon. The vision encoder is preserved (full multimodal).

Orion is TheDrummer's finetune of google/gemma-4-26B-A4B-it — a 25.2B-parameter MoE (Mixture of Experts) model with 128 experts per layer (avg. 3.8B active parameters per token), tuned in his roleplay/creative-writing lineage. The unquantized BF16 source was used for maximum fidelity.

BaseQuant_XL keeps the most routing-critical layers in full bf16 precision — the MoE router.proj (gate) and the mlp (shared/dense expert that processes all tokens) — while applying aggressive quantization to the bulk parameters. tie_word_embeddings=True means there is no separate lm_head (tied to embed_tokens).

About XL Quantization

BaseQuant_XL is a fully data-agnostic, static quantization. No calibration dataset, no sensitivity analysis, no importance matrix. Precision is allocated purely by architectural role — routing-critical layers get higher precision, bulk expert parameters get lower precision. The result is a transparent, faithful capture of the source model.

Data-dependent calibration quantizations (iMatrix, AWQ, GPTQ, oQ, oQ4e, etc.) use a calibration set to guide bit allocation. This can produce a skewed representation of the model: domains well-represented in the calibration data (English, popular topics, public or leaked benchmarks) are preserved better, while underrepresented domains (non-English languages, niche use cases, your own data) are preserved worse. XL avoids this trade-off entirely — it generalizes honestly because it is never fit to any particular data distribution.

Use with mlx

bash

pip install -U mlx-vlm

bash

python -m mlx_vlm.generate --model leonsarmiento/Orion-26B-A4B-v1-6bit-XL-mlx --max-tokens 256 --temperature 1.0 --top-p 1.0 --prompt "Hello"

BaseQuant_XL Quantization Strategy

Bit Depth	Layers	Rationale
bf16 (unquantized)	`router.proj` (gate), `mlp` (shared/dense expert), vision tower	Routing decisions and the dense expert that processes every token — errors here cascade through all tokens
8-bit	`embed_tokens`, `self_attn`	Every-token layers with moderate sensitivity — 8-bit is near-lossless
6-bit	`experts.switch_glu` (routed experts)	Bulk of parameters — only a subset of experts active per token

Quantization Details

Layer	Bits	Group Size
`router.proj` (gate)	bf16	—
`mlp` (shared/dense expert)	bf16	—
Vision tower (full)	bf16	—
`embed_tokens`	8	64
`self_attn`	8	64
`experts.switch_glu` (routed experts)	6	64
Default fallback	8	64

—Quantization type: BaseQuant_XL mixed (multimodal, vision preserved)
—Bits per weight: ~7.1
—Total size: 22.8 GB (5 shards)
—Group size: 64
—Method: Custom quant_predicate via mlx_vlm
—Note: tie_word_embeddings=True — no separate lm_head

Recommended Inference Parameters

Parameter	Value
`temperature`	1.0
`top_p`	1.0
`top_k`	64
`min_p`	0.01
`repeat_penalty`	1.1

Reasoning Parsing in LM Studio and Others

—Start string: <|channel|>thought
—End string: <|channel|>

Chat Template

The chat template is synced to the canonical Google upstream version from `google/gemma-4-26B-A4B-it/chat_template.jinja`, replacing the template shipped with the source finetune. Do not rely on the HF cache copy — it may be stale.

Key markers in the latest template: preserve_thinking, continues_into_next, prev_non_tool_role, null handling in format_argument.

The template is injected into both chat_template.jinja (standalone file) and tokenizer_config.json (chat_template JSON field).