majentik/MERaLiON-3-10B-MLX-4bit

MERaLiON-3-10B-MLX-4bit

4-bit MLX quantization of MERaLiON/MERaLiON-3-10B, the instruction-tuned Singaporean multilingual speech-language model from A\STAR / I²R — in a loadable single-tree layout* with a working MLX inference harness.

2026-07-07 republish: this repo previously shipped a split component layout (encoder.safetensors / adaptor.safetensors / decoder-*.safetensors) that no MLX runner could load directly. It now ships the standard model.safetensors.index.json tree produced by pipelines/mlx_direct_quantize, verified with audio transcription through the MLX harness below.

Quality caveat (4-bit): output stays coherent but is measurably less faithful than the 8-bit variant: on the code-switched Singlish sample it paraphrases and translates the Mandarin tail instead of transcribing it verbatim (There's like two quarters, ma. Then he will teach. vs 8-bit's there's like two quarters ma then 他会教). For faithful code-switched ASR prefer the 8-bit pack; this variant suits lower-memory speech understanding (summaries, Q&A over audio).

At a glance

Unlike the 3B-ASR sibling, MERaLiON-3-10B mixes its speech encoder's per-layer outputs with learned softmax weights (speech_config.use_weighted_layer_sum=true); the 1-D weight vector rides through unquantized with the rest of the speech tower.

Verification (2026-07-07, M-series Mac, 128 GB)

• —Composite-load self-test: PASS (decoder quantized, speech tower bf16, finite logits, decode bitwise-matches the mlx_lm gemma2 forward).
• —Audio transcription on the upstream MERaLiON sample clips runs end to end and stays coherent, with the fidelity caveat above.

Usage

Runs on the MERaLiON-3 MLX harness in `majek/pipelines/meralion3_mlx` (Whisper encoder + weighted layer sum + adaptor in MLX, Gemma-2 decoder via mlx_lm.models.gemma2):

# one-shot transcription
python -m pipelines.meralion3_mlx.generate \
    --path majentik/MERaLiON-3-10B-MLX-4bit \
    --wav your_clip.wav

# local HTTP endpoint (POST wav bytes or multipart "file")
python -m pipelines.meralion3_mlx_serve \
    --model majentik/MERaLiON-3-10B-MLX-4bit --port 8321

Stock mlx-lm alone cannot load this model (composite audio architecture, model_type=meralion3); the harness routes the speech tower and decoder from the same checkpoint tree. The prompt template is the upstream one: Instruction: <TextHere> \nFollow the text instruction based on the following audio: <SpeechHere> inside Gemma-2 chat turns.

Quantization recipe

Only the Gemma-2-9B decoder's 2-D tensors are quantized (where >95 % of parameters live). The Whisper encoder, speech adaptor, layer-sum weights, norms, embeddings and all 1-D tensors are preserved bf16 — they are small and disproportionately sensitive to quantization noise.

Reproduce:

python -m pipelines.mlx_direct_quantize --model meralion3-10b \
    --base-dir /tmp/mlx-direct-release/meralion3-10b/base \
    --out-dir /tmp/mlx-direct-release/meralion3-10b/4bit \
    --bits 4 --mode affine --group-size 64

Intended use

Speech understanding for Singapore-context English, Mandarin, Malay, Tamil (plus Indonesian, Thai, Vietnamese) locally on Apple Silicon, where the 8-bit pack does not fit alongside other workloads.

Licence

Derivative of MERaLiON-3-10B under the MERaLiON-3-Public-Licence, which permits derivatives with notices. This repository redistributes quantized weights with the required attribution; upstream model, training data and evaluation details are documented on the base model card.