MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit

Cohere Transcribe 03-2026 — MLX Mixed 2-bit/3-bit/4-bit

A dynamically quantized MLX build of CohereLabs/cohere-transcribe-03-2026: instead of one bit-width per section, every module gets a bit-width assigned from an empirical quantization-sensitivity map (Unsloth-style dynamic quantization, adapted to ASR). Most of the encoder runs at 2-bit; the empirically fragile groups (decoder, last encoder blocks) stay at 3/4-bit.

Compared to the uniform 3-bit/4-bit build, this checkpoint is 14% smaller and ~15% faster at nearly the same WER.

Key Metrics

Method: sensitivity-guided bit allocation

1. 1.Sensitivity scan — each (4-block, component) group of the 48-layer FastConformer encoder is fake-quantized to 2-bit alone (rest FP16) and scored by transcript divergence (CER) against the FP16 teacher on a calibration set.
2. 2.Greedy allocation — groups above sensitivity thresholds are held at 4-bit / 3-bit; everything else drops to 2-bit (group size 64, affine). The decoder — the single most sensitive component — stays at 4-bit.
3. 3.The resulting per-module rule map ships in this repo as `quant_bitmap.json` for full reproducibility.

Uniform 2-bit costs +2.01 pt WER; this allocation recovers ~75% of that damage for only +0.4 effective bits.

1x1 Conv1d layers are converted to Linear equivalents to enable quantization.

Architecture

• —Base model: Cohere Transcribe 03-2026 (~2B params)
• —Encoder: FastConformer (48 layers, d=1280)
• —Decoder: Transformer (8 layers, d=1024)
• —Tokenizer: SentencePiece (16,384 tokens)

Usage

Requires mlx-audio installed from git main:

pip install "mlx-audio[stt] @ git+https://github.com/Blaizzy/mlx-audio.git"

# apply the loader patch shipped in this repo (fixes two upstream
# quantized-reload bugs; see mlx_audio_cohere_quant_patch.py)
from mlx_audio_cohere_quant_patch import apply_patch
apply_patch()

from mlx_audio.stt import load

model = load("MarkChen1214/cohere-transcribe-03-2026-MLX-Mixed-2bit3bit4bit")
result = model.generate(audio="audio.wav", language="en", punctuation=True)
print(result.text)

Eval Results (Full LibriSpeech, seed=42)

Evaluated with the HF Open ASR Leaderboard methodology (Whisper EnglishTextNormalizer + jiwer).

License

MIT — use it however you like, attribution appreciated.

The base model (CohereLabs/cohere-transcribe-03-2026) is Apache 2.0; its notices apply to the underlying weights.