efederici/parakeet-tdt-0.6b-v3-onnx-int4

Parakeet TDT 0.6B v3 — ONNX Quantized (int4/int8 hybrid)

Quantized ONNX export of nvidia/parakeet-tdt-0.6b-v3 for browser and edge inference with parakeet.js.

409 MB total — 6x smaller than fp32, 39% smaller than istupakov/int8, with half the quantization degradation and 17% faster inference.

Files

Usage with parakeet.js (browser)

import { fromUrls } from 'parakeet.js';

const BASE = 'https://huggingface.co/efederici/parakeet-tdt-0.6b-v3-onnx-int4/resolve/main';

const model = await fromUrls({
  encoderUrl: `${BASE}/encoder-model.int4.onnx`,
  decoderUrl: `${BASE}/decoder_joint-model.int8.onnx`,
  tokenizerUrl: `${BASE}/vocab.txt`,
  preprocessorBackend: 'js',
  backend: 'webgpu', // or 'wasm'
});

const result = await model.transcribe(pcm, 16000, {
  returnTimestamps: true,
  returnConfidences: true,
});
console.log(result.utterance_text);

For long recordings:

const result = await model.transcribeLongAudio(pcm, 16000, {
  returnTimestamps: true,
  chunkLengthS: 95,
});
console.log(result.text);
console.log(result.chunks);

Benchmark

LibriSpeech test-clean (200 samples, 26 min of audio)

CPU inference with onnxruntime.

Quality (WER vs ground truth, lower is better)

All three models achieve the same ground-truth WER (~1.7%). The quantized models actually score marginally better due to a slight regularization effect.

Quantization degradation (WER vs fp32 output)

Half the degradation of int8, at 39% smaller size and 17% faster speed.

Individual samples (median of 5 runs)

Speed (RTF = processing time / audio duration, lower is better)

Quality (WER vs fp32 reference)

Quantization details

Hybrid approach for optimal size/quality/speed:

• —Encoder pointwise Conv layers converted to MatMul for better int4 coverage (using onnx-conv2matmul)
• —Encoder linear + pointwise Conv (87.5% of weights): int4 MatMulNBits, block_size=64, asymmetric
• —Encoder depthwise Conv (small): fp32
• —Decoder (LSTM + embedding + linear): int8 dynamic quantization
• —Compatible with ONNX Runtime (CPU, WASM, WebGPU)

Source fp32 model: istupakov/parakeet-tdt-0.6b-v3-onnx

License

CC-BY-4.0, inherited from nvidia/parakeet-tdt-0.6b-v3.