Alkd/Qwen3-ASR-1.7B-audio8-text4-mlx

Qwen3-ASR-1.7B — Mixed-Precision MLX (audio-8bit / text-4bit)

A mixed-precision quantized version of Qwen3-ASR-1.7B for Apple Silicon (MLX framework).

This configuration was identified as optimal after a systematic ablation study across 16 quantization configurations on Common Voice 22.0 (zh-TW / en / ja / ko).

Performance (n=1000, Common Voice 22.0)

Evaluated against `mlx-community/Qwen3-ASR-1.7B-4bit` on the same benchmark under identical conditions.

Verdict: Accuracy is statistically equivalent to mlx-community/Qwen3-ASR-1.7B-4bit. The model is 18% smaller on disk and ~10% faster at inference.

Note: Japanese (ja) and other languages with embedtokens quantized to 4-bit exhibit hallucination (~172% CER). This is the same behaviour as `mlx-community/Qwen3-ASR-1.7B-4bit` (174% CER) and is an inherent limitation of 4-bit embedtokens quantization, not specific to this model.

Why a custom loader is needed

mlx-audio's load_model hardcodes a model_quant_predicate inside Qwen3ASRForConditionalGeneration that unconditionally prevents the audio_tower from being quantized:

# mlx_audio/stt/models/qwen3_asr.py (~line 779)
def model_quant_predicate(self, path, module, group_size, bits):
    return not path.startswith("audio_tower")

Because of this guard, calling load_model(repo_id) on this repo would silently load the 8-bit audio weights into un-quantized Linear layers, causing a shape mismatch and producing empty transcriptions.

The workaround is to load the bf16 base model, apply quantization manually (bypassing model_quant_predicate), then replace the weights with the saved ones. The complete loader is shown below.

Usage

Requirements

pip install mlx-audio>=0.3.1 mlx>=0.22

Transcription

import mlx.core as mx
import mlx.nn as nn
from mlx_audio.stt.utils import load_model
from mlx_audio.stt.generate import generate_transcription

# ── Step 1: load bf16 base ────────────────────────────────────────────────────
bundle = load_model("mlx-community/Qwen3-ASR-1.7B-bf16")
model  = bundle._model

# ── Step 2: apply the same mixed-precision quantization ──────────────────────
def pred_audio(path, m):
    if not isinstance(m, (nn.Linear, nn.Embedding)):
        return False
    return path.startswith("audio_tower.")

def pred_text(path, m):
    if not isinstance(m, (nn.Linear, nn.Embedding)):
        return False
    return path.startswith("model.") and not path.startswith("model.audio_tower")

nn.quantize(model, group_size=64, bits=8, class_predicate=pred_audio)
nn.quantize(model, group_size=64, bits=4, class_predicate=pred_text)

# ── Step 3: load the saved weights from this repo ────────────────────────────
from huggingface_hub import hf_hub_download

weights_path = hf_hub_download("Alkd/Qwen3-ASR-1.7B-audio8-text4-mlx", "model.safetensors")
saved = list(mx.load(weights_path).items())
model.load_weights(saved, strict=True)
mx.eval(model.parameters())

# ── Step 4: transcribe ────────────────────────────────────────────────────────
# language=None  → auto-detect (recommended for zh-TW / Chinese)
# language="English", "Japanese", "Korean", etc. → force language
result = generate_transcription(bundle, "audio.wav", language=None)

# generate_transcription returns either a str or an STTOutput object
if isinstance(result, str):
    text = result
elif hasattr(result, "text"):
    text = result.text or ""
else:
    text = str(result)

print(text)

Language hints

How this model was created

Quantization procedure

import mlx.nn as nn

def pred_audio(path, m):
    return isinstance(m, (nn.Linear, nn.Embedding)) and path.startswith("audio_tower.")

def pred_text(path, m):
    return (isinstance(m, (nn.Linear, nn.Embedding))
            and path.startswith("model.")
            and not path.startswith("model.audio_tower"))

nn.quantize(model, group_size=64, bits=8, class_predicate=pred_audio)  # audio tower → 8-bit
nn.quantize(model, group_size=64, bits=4, class_predicate=pred_text)   # LM layers + embed → 4-bit

Architecture equivalence

The text+embed component (model.layers + model.embed_tokens) is quantized identically to mlx-community/Qwen3-ASR-1.7B-4bit. Running our pred_text predicate alone on the bf16 base model reproduces the mlx-community weights digit-for-digit (verified across all four languages on the same benchmark).

The only difference is the audio_tower: mlx-community keeps it in bf16 (3,580 MB → 1,529 MB), while this model quantizes it to 8-bit (→ 1,253 MB), saving an additional 276 MB.

Component sizes

Ablation study summary (n=250 per language)

The configuration was selected from 16 candidates evaluated on Common Voice 22.0 (zh-TW, en, ja, ko):

† Japanese CER methodology: 2 out of 250 samples trigger an infinite repetition loop (e.g. "イギリスの" × 700 times for a 58-char reference), inflating the raw aggregate CER to ~163–174%. The reported figures use a 3× length cap (hypothesis truncated to min(len(hyp), len(ref)×3)), which correctly bounds the loop penalty. The remaining 248/250 samples show 32.4% CER — nearly identical to the bf16 baseline (31.91%). Both this model and mlx-community/Qwen3-ASR-1.7B-4bit exhibit the same loop behaviour; adding repetition_penalty=1.1 or a max_tokens limit at inference time eliminates the issue entirely.

Limitations

• —Japanese repetition loop: 4-bit embed_tokens causes 2/250 samples to enter an infinite repetition loop, inflating raw aggregate CER to ~163%. On the 248/250 unaffected samples the CER is 32.4% (comparable to bf16 baseline). The same issue exists in mlx-community/Qwen3-ASR-1.7B-4bit. Mitigation: pass repetition_penalty=1.1 or a max_tokens cap to generate_transcription.
• —Custom loader required: See Why a custom loader is needed above.
• —MLX only: Weights are in MLX safetensors format and are not compatible with PyTorch/Transformers.

Future Work / TODO

The current quantization scheme applies a uniform bit-width per component (all audio_tower layers get 8-bit, all text layers get 4-bit). This is a coarse-grained assignment. The following directions would yield a better Pareto frontier:

1. Per-layer sensitivity analysis (HAWQ-style)

HAWQ uses the Hessian trace of each layer's loss contribution as a sensitivity proxy. Layers with high Hessian trace are more sensitive to quantization and should receive more bits; insensitive layers can be pushed lower.

How to apply here: run a forward+backward pass on a small calibration set, compute Tr(H_i) per layer, then use the ranking to assign bit-widths via class_predicate. This replaces the current hand-tuned audio-8 / text-4 split with a data-driven one.

Possible outcome: some audio_tower middle layers could be dropped to 4-bit (saving ~80–120 MB) while keeping early/late transformer layers and the CNN frontend at 8-bit.

2. Better within-layer quantization (AWQ-style)

AWQ improves quality at a fixed bit-width by identifying per-channel "salient" weights (those corresponding to large activation magnitudes) and applying a learned per-channel rescaling before quantization. The scale factors are absorbed into adjacent layers so inference cost is unchanged.

Distinction from HAWQ: AWQ answers how to quantize a layer better; HAWQ answers which bit-width each layer should get. They are complementary and can be stacked.

Implementation note: AWQ requires scale absorption into paired layers (e.g. Linear → LayerNorm), which needs architecture-aware code beyond MLX's built-in nn.quantize. Non-trivial to implement for the audio CNN + transformer hybrid architecture.

3. embed_tokens at 8-bit (Japanese hallucination fix)

Keeping embed_tokens at 8-bit instead of 4-bit (inspired by Unsloth's dynamic quantization findings) would add ~155 MB but is predicted to eliminate the Japanese hallucination:

This would still be ~8% smaller than mlx-community/Qwen3-ASR-1.7B-4bit while restoring Japanese quality.

Citation

If you use this model, please cite the original Qwen3-ASR work and mlx-audio:

@misc{qwen3asr2025,
  title  = {Qwen3-ASR},
  author = {Qwen Team},
  year   = {2025},
  url    = {https://huggingface.co/Qwen/Qwen3-ASR-1.7B}
}

License

Apache 2.0 — same as the base model.