sahilchachra/sarvam-30b-MXFP4

sarvam-30b — MLX MXFP4

MLX MXFP4 quantization of `sarvamai/sarvam-30b`, a 32B-parameter Mixture-of-Experts text-generation model from Sarvam AI. Text-only (no vision tower).

sarvam-30b uses a custom `sarvam_moe` architecture: 19 transformer layers (the first dense, the rest MoE), 128 routed experts + 1 shared expert with top-6 sigmoid-gated routing (DeepSeek-style: expert bias for load balancing, routed_scaling_factor 2.5), grouped-query attention (64 query heads / 4 KV heads) with QK-RMSNorm, and an untied 262K-vocab lm_head (multilingual tokenizer covering Indian languages). Runs on Apple Silicon via mlx-lm.

⚠️ Requires vendoring an unmerged mlx-lm PR. As of this quantization, sarvam_moe is not in any released `mlx-lm` version — support exists only as an open, unmerged draft, ml-explore/mlx-lm#991. This repo's weights were produced using that PR's model definition, vendored locally. To load this model you must vendor the same file into your own mlx-lm install:

pip install -U mlx-lm
curl -o "$(python -c 'import mlx_lm, os; print(os.path.dirname(mlx_lm.__file__))')/models/sarvam_moe.py" \
  https://raw.githubusercontent.com/ml-explore/mlx-lm/c686bdab1cbf6b5f5364675a8bc858ead97c92eb/mlx_lm/models/sarvam_moe.py

Once PR #991 merges, this step won't be necessary. This will not load in LM Studio (or any tool bundling its own unpatched mlx-lm/mlx-vlm) until then.

MoE router weight (mlp.gate.weight) — quantizing the router double-damages routing decisions, so the architecture's own quantization predicate excludes it |

Quantizations

Verification

Smoke-tested on Apple Silicon via mlx-lm with deterministic greedy decoding, inspecting raw token IDs (not just detokenized text) — this matters extra here because embed_tokens (the input embedding table) is quantized, unlike lm_head which the architecture's own quantization predicate keeps full precision.

Tested English, Hindi, and Tamil (this is a multilingual Indian-language model, so an English-only check would miss degradation in non-Latin scripts):

All raw token-ID sequences were clean (no repetition loops or garbage runs).

Usage (mlx-lm)

pip install -U mlx-lm
# then vendor mlx_lm/models/sarvam_moe.py from PR #991 as shown above

from mlx_lm import load, generate

model, tokenizer = load("sahilchachra/sarvam-30b-MXFP4")
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=256, verbose=True))

This is a reasoning model; it emits a <think>...</think> block before its answer — give it enough max_tokens or the visible answer can be truncated while it's still reasoning.

Notes & limitations

• —Custom architecture, unmerged upstream support. See the vendoring step above — this is a hard requirement, not a suggestion, until PR #991 merges.
• —Text-only. No vision/image support in the base model.
• —tie_word_embeddings: false — lm_head is a separate ~1.07B-parameter matrix, kept in bf16 (see table above).
• —embed_tokens (the input embedding table) is quantized. The MXFP4 smoke test above specifically checked non-English prompts to catch degradation here.
• —Inherits all capabilities and limitations of the base model. See the original model card.
• —Quantized by @sahilchachra with MLX. Apache-2.0.