MarkChen1214/nemotron-3.5-asr-streaming-0.6b-MLX-Mixed-6bit8bit

Nemotron 3.5 ASR Streaming 0.6B — MLX Mixed 6-bit/8-bit

Sensitivity-guided mixed-precision MLX conversion of `mlx-community/nemotron-3.5-asr-streaming-0.6b`, which is the unquantized BF16 MLX conversion of `nvidia/nemotron-3.5-asr-streaming-0.6b`.

This mixed 6-bit/8-bit profile uses affine 6-bit weights by default and protects the RNN-T joint, language-prompt projection, and final encoder FFN1 hotspots at 8-bit. Group size is 64. Convolution and normalization weights that are not MLX-quantizable remain BF16.

The result matches the MLX Community uniform-8-bit quality and throughput class on the full 428-sample FLEURS-Ar test set while reducing total artifact size by 16.75%.

Results

Measured locally on an Apple M4 Pro (14-core, 48 GB), MLX 0.32.0 and mlx-audio 0.4.5 from git main. All models use the best-quality 1,120 ms look-ahead ([56, 13]) and the same repository evaluator and normalizers.

Full FLEURS-Ar test set

The 0.08 WER and 0.02 CER differences are a practical tie. The defensible result is equal full-set quality and throughput with an exact 120.83 MiB / 16.75% size reduction.

Matched fast screens

English is the macro-average of LibriSpeech clean, LibriSpeech other, AMI and Earnings-22. Casablanca is the macro-average of eight Arabic dialects. Each dataset uses 25 samples selected with seed 42; one empty AMI reference leaves 24 scored samples.

Fast-screen sub-point differences are not statistically significant. They are included to show the complete compression frontier and the controls tested during development, not to claim accuracy superiority over BF16 or uniform 8-bit.

How the mixed allocation was built

The source is pinned to BF16 revision e550040c0478027ed679b2b6b0d055502c103663.

1. 1.Six FLEURS-Ar and six LibriSpeech-clean clips were selected with seed 42.
2. 2.Each encoder component group and transducer region was independently fake-quantized to 2-bit while the rest of the model stayed BF16.
3. 3.Transcript CER against the BF16 teacher identified the RNN-T joint (109.15%), final encoder FFN1 (96.32%), and prompt projection (86.72%) as the dominant hotspots.
4. 4.The performance bitmap uses 6-bit affine weights for 210 quantized modules and 8-bit weights for 13 protected modules.
5. 5.The final artifact was strict-loaded and evaluated end to end; no runtime monkeypatch is required.

The exact allocation is included as quant_bitmap.json. Conversion and scan code, rejected ablations, per-dataset measurements, and the external artifact audit are maintained in `yuting1214/model-lab`.

Usage

Nemotron support currently requires mlx-audio from git main:

pip install "git+https://github.com/Blaizzy/mlx-audio.git"

from mlx_audio.stt import load

model = load(
    "MarkChen1214/nemotron-3.5-asr-streaming-0.6b-MLX-Mixed-6bit8bit",
    strict=True,
)

result = model.generate(
    "speech.wav",
    language="ar",
    punctuation=True,
    att_context_size=[56, 13],
)
print(result.text)

To reproduce the conversion from the pinned BF16 source, first download the published allocation file:

hf download MarkChen1214/nemotron-3.5-asr-streaming-0.6b-MLX-Mixed-6bit8bit quant_bitmap.json \
  --local-dir /tmp/nemotron-mixed-6bit8bit-spec

python 01-local-inference/convert_nemotron_asr_mixed.py \
  --bitmap /tmp/nemotron-mixed-6bit8bit-spec/quant_bitmap.json \
  --output /tmp/nemotron-mixed-6bit8bit

Intended use and limitations

• —The architecture is natively cache-aware and streaming-capable, but the measurements above use whole-utterance inference. They do not establish p50/p99 chunk latency, partial-transcript stability, or finalization latency.
• —Full FLEURS-Ar is complete. English and Casablanca values above are screening slices; full LibriSpeech and the larger Casablanca gate remain pending.
• —Nemotron is not the Arabic offline-accuracy leader in this project. Its differentiated product value is multilingual native streaming.
• —Accuracy varies by language, dialect, acoustic condition, and look-ahead. Evaluate on your own target distribution before deployment.

License

The original NVIDIA model is released under the NVIDIA Open Model License. This quantized derivative inherits those terms. Review the license for your intended use and retain the required attribution.