visuall/parakeet-tdt-0.6b-v3-onnx-fp16

parakeet-tdt-0.6b-v3-onnx-fp16

Half-precision ONNX export of [nvidia/parakeet-tdt-0.6b-v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), converted from the fp32 ONNX export published by [istupakov/parakeet-tdt-0.6b-v3-onnx](https://huggingface.co/istupakov/parakeet-tdt-0.6b-v3-onnx).

Published because upstream ships only int8 and fp32. The int8 build costs real accuracy, and fp32 costs 2.5 GB for no measurable gain over fp16.

Files

Model inputs and outputs stay fp32 (keep_io_types=True), so this is a drop-in replacement for the fp32 export — only the internal weights are half precision.

Why not int8

Measured against human reference transcripts, same pipeline, same audio:

Per-clip sign test on FLEURS: French p=2·10⁻¹⁰, Spanish p=9·10⁻⁶.

fp16 vs fp32 across the AMI pool: 6 word edits out of 13 638 (0.04%), WER identical to three decimals — so fp32 is not worth twice the size.

Cost of fp16 over int8: 1.28 GB vs 0.67 GB, and ~15% slower inference (RTF 0.077 vs 0.067, measured on Apple Silicon via ONNX Runtime + CoreML).

Conversion recipe

from onnxconverter_common import float16
import onnx

# Large-model path: the fp32 encoder keeps its weights in encoder-model.onnx.data,
# so the in-memory converter cannot run shape inference on it.
m = float16.convert_float_to_float16_model_path(
    "encoder-model.onnx", keep_io_types=True
)
onnx.save(m, "encoder-model.fp16.onnx")

⚠️ Required patch. The converter leaves three Cast nodes in /pre_encode with to=FLOAT while their consumers become fp16. Without fixing them ORT refuses to load the graph:

Type Error: Type parameter (T) of Optype (Add) bound to different types
            (tensor(float) and tensor(float16))

from onnx import TensorProto
g = onnx.load("encoder-model.fp16.onnx")
outs = {o.name for o in g.graph.output}
for n in g.graph.node:
    if n.op_type == "Cast" and n.output[0] not in outs:
        for a in n.attribute:
            if a.name == "to" and a.i == TensorProto.FLOAT:
                a.i = TensorProto.FLOAT16
onnx.save(g, "encoder-model.fp16.onnx")

The decoder converts cleanly with the in-memory convert_float_to_float16.

License and attribution

CC-BY-4.0, inherited from nvidia/parakeet-tdt-0.6b-v3. Attribution: NVIDIA for the model, istupakov for the ONNX export this build was converted from.