CoolFace
Modelpublic

pierrelamy/sarvam-105b-mlx-q6

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
0likes35downloads
Model Card

Sarvam-105B MLX Q6

This is an MLX 6-bit affine quantization of sarvamai/sarvam-105b, converted from the public BF16 checkpoint abhinand/sarvam-105b-bf16.

The model is a base/pretrain text-generation model using the sarvam_mla architecture: DeepSeek-V3-style MLA attention plus sigmoid MoE routing. It is not an instruction-tuned chat model.

Why Q6

We tested both q4 and q6 locally on Apple Silicon. The q4 conversion loads and runs, but it is not usable: it collapses to whitespace with probability 1.0 across sanity prompts. The q6 conversion is coherent and passed real-weight validation, so q6 is the published practical checkpoint.

Local validation summary:

QuantSizeResult
q4, group size 6456 GBBroken: collapsed whitespace forward
q6, group size 6480 GBCoherent top-token and generation checks

Example q6 top-token probes:

PromptTop prediction
The capital of France is Paris
def add(a, b):\n return a

Validation

Measured locally on an Apple Silicon M5 Max host with 128 GB unified memory.

  • —Real BF16 source checkpoint loaded through the MLX sarvam_mla port during conversion.
  • —q6 generation smoke passed.
  • —Warm short-context decode measured about 46 tok/s, with about 86 GB peak memory.
  • —Long-context absorbed-MLA checks ran at 2k, 8k, and 16k context.
  • —Prefix-cache and prompt-lookup decoding checks passed on suitable workloads.

The q4 collapse was investigated against RoPE, config mapping, and the kv_b_proj sanitize path; those matched the reference implementation. The failure is attributed to naive 4-bit quantization of this large DeepSeek-style MLA-MoE model's activation outliers.

Loading

This artifact includes the custom configuration/modeling files referenced by auto_map. Use MLX tooling with remote/local custom code support as needed.

python
from mlx_lm import load, generate

model, tokenizer = load("pierrelamy/sarvam-105b-mlx-q6")
response = generate(model, tokenizer, prompt="The capital of France is", max_tokens=32)
print(response)

Provenance

  • —Base model: sarvamai/sarvam-105b
  • —BF16 conversion source: abhinand/sarvam-105b-bf16
  • —Format: MLX safetensors
  • —Quantization: affine 6-bit, group size 64
  • —Tensor payload reported by model.safetensors.index.json: 86,202,965,504 bytes
  • —Parameters reported by model.safetensors.index.json: 106,031,767,424

Caveats

  • —This is a base model, not an instruct model.
  • —q4 is intentionally not published because it collapsed in local validation.
  • —The checkpoint is large: expect roughly 80 GB on disk and about 86 GB peak memory in the measured short-context generation setup.