lwittich/parakeet-tdt-0.6b-v3-onnx-int8-cpu

Parakeet TDT 0.6B v3, ONNX INT8

This repository provides an ONNX Runtime compatible, selectively INT8-quantized variant of NVIDIA's nvidia/parakeet-tdt-0.6b-v3 automatic speech recognition model.

The model was prepared for CPU-based inference under constrained local deployment conditions. The intended use case is low-latency multilingual ASR in local speech-processing pipelines where inference cost, memory use, and real-time behaviour are important deployment constraints.

This is not an official NVIDIA release.

Attribution

This model is derived from NVIDIA's Parakeet TDT 0.6B v3.

Quickstart

pip install onnxruntime onnx-asr[cpu,hub]

import onnx_asr

model = onnx_asr.load_model(
    "<REPO_ID>",
    providers=["CPUExecutionProvider"],
)

text = model.recognize(audio_array)
print(text)

Input audio must be:

• —mono
• —16 kHz
• —float32 NumPy array

Quantization

Dynamic quantization was performed using ONNX Runtime. Quantization is applied selectively to matrix multiplication operators in order to reduce model size and CPU inference cost while preserving the original convolution operators.

Quantized operators

• —MatMul
• —Gemm

Excluded operators

• —Conv

Rationale

Dynamic INT8 weight quantization reduces the storage and memory bandwidth requirements of matrix multiplication weights while keeping activations in FP32. Convolution operators were left unquantized because quantizing them increased runtime overhead and degraded transcription quality in the tested CPU execution environment.

Benchmark Summary

Evaluation setup

WER was computed after:

• —Unicode NFKC normalization
• —lowercasing
• —punctuation removal
• —whitespace normalization

RTF is computed as:

RTF = inference_time_seconds / audio_duration_seconds

Values below 1.0 indicate faster-than-real-time inference.

Repository structure

.
├── encoder-model.onnx
├── encoder-model.onnx.data
├── decoder_joint-model.onnx
├── decoder_joint-model.onnx.data
├── config.json
├── quantization_config.json
├── vocab.txt
└── README.md

Intended use

This model is intended for research and development involving local multilingual ASR, CPU inference, and constrained deployment studies. It may be useful as a component in cascaded speech-to-text and text-translation pipelines.

Limitations

• —This is not an official NVIDIA release.
• —The model is derived from the original NVIDIA Parakeet TDT 0.6B v3 model and inherits its language coverage and modelling limitations.
• —The quantization is selective and weight-only. It should not be interpreted as full INT8 inference.
• —Runtime performance depends on CPU architecture, ONNX Runtime version, threading configuration, and memory bandwidth.
• —The benchmark results are specific to the stated evaluation setup and should not be assumed to generalize to all devices.
• —The model performs ASR only. It does not perform speech translation or text translation by itself.

License

This repository follows the original model license: CC-BY-4.0.

The original model is © NVIDIA Corporation. This repository contains a converted and quantized derivative artifact.

Citation

@misc{wittich2026parakeet_tdt_onnx_int8,
  title        = {Parakeet TDT 0.6B v3 ONNX INT8},
  author       = {Wittich, Lucas},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/<REPO_ID>}},
  note         = {ONNX Runtime compatible selectively INT8-quantized derivative of NVIDIA Parakeet TDT 0.6B v3}
}