Brooooooklyn/Gemma-4-26B-A4B-IT-UD-Q8_K_XL-mlx

Gemma-4-26B-A4B-IT — UD-Q8KXL (mlx-node)

8-bit affine quantization of google/gemma-4-26b-a4b-it for Apple Silicon, using the **Unsloth Dynamic** quantization strategy via mlx-node.

All Variants

Benchmarked on Apple M3 Max 128GB via `examples/lm.ts` (best decode tok/s across turns 2–4, steady-state, capitals chat with reasoningEffort: 'low').

Note: No Q2 variant is published — Gemma-4-26B-A4B-IT has only ~4B active parameters per token, which is below the architectural redundancy needed for 2-bit quantization to remain coherent. Both unsloth and mixed_2_6 recipes produced gibberish at Q2 on this model.

Performance

Steady-state decode: 49.8 tok/s on Apple M3 Max 128GB (best of turns 2–4, examples/lm.ts capitals chat with reasoningEffort: 'low'). Decode is memory-bandwidth bound on Apple Silicon — fewer bytes per token directly translates to higher throughput. The MoE architecture activates only top-K of 128 experts per token (~4B active out of ~26B total), and the compiled C++ forward graph fuses the per-layer dispatch.

Per-Tensor Bit Assignments (N=8)

Quantization Strategy

Built on Unsloth Dynamic 2.0 per-tensor KLD analysis. At --q-bits 8 the unsloth recipe assigns the base bits to MLP gate/up projections (the bulk of the parameter budget), base+1 to downproj (slightly more sensitive), `base+2` (snapped to a valid bit width) + AWQ pre-scaling to attention q/k/v projections, `base+2` to `embedtokens, base+3 (capped/snapped) to the routing-critical paths, and keeps selfattn.oproj as bf16 (AWQ-uncorrectable — its inputs come from the attention compute, not from a norm layer). The MoE router (router.proj`) is forced to 8-bit affine to preserve top-K expert selection accuracy.

imatrix AWQ pre-scaling amplifies important weight channels and fuses inverse scales into preceding layer norms (zero inference overhead).

Architecture

Usage

import { loadSession } from '@mlx-node/lm';

const session = await loadSession('./Gemma-4-26B-A4B-IT-UD-Q8_K_XL-mlx');

for await (const event of session.sendStream('Explain the MoE architecture in Gemma-4.', {
  config: { maxNewTokens: 2048, temperature: 0.6, reasoningEffort: 'low' },
})) {
  if (!event.done) process.stdout.write(event.text);
}

How It Was Made

mlx convert \
  -i gemma-4-26b-a4b-it \
  -o Gemma-4-26B-A4B-IT-UD-Q8_K_XL-mlx \
  -q --q-bits 8 --q-recipe unsloth \
  --imatrix-path imatrix_unsloth.gguf

Acknowledgments

• —[Unsloth](https://unsloth.ai) — Quantization strategy based on their per-layer KLD benchmarks and Dynamic 2.0 methodology
• —[Google DeepMind](https://deepmind.google/) — For the Gemma-4 model family
• —[Apple MLX](https://github.com/ml-explore/mlx) — For the Metal-accelerated ML framework

License

Gemma Terms of Use (inherited from base model).