younghan-meta/Parakeet-TDT-ExecuTorch-CUDA-Windows-Quantized
Parakeet-TDT-ExecuTorch-CUDA-Windows-Quantized
Pre-exported ExecuTorch .pte file for Parakeet TDT 0.6B with CUDA-Windows backend (NVIDIA GPU), bf16 precision, 4-bit weight linear quantization, and 8-bit weight embedding quantization.
3.2× smaller and 2.5× faster prefill than the fp32 variant with identical word-level transcription accuracy.
For the non-quantized variant, see Parakeet-TDT-ExecuTorch-CUDA-Windows.
Quantization Details
Installation
git clone https://github.com/pytorch/executorch/ ~/executorch
cd ~/executorch && pip install .Build on Windows (PowerShell):
cmake --workflow --preset llm-release-cuda
Push-Location examples/models/parakeet
cmake --workflow --preset parakeet-cuda
Pop-LocationDownload
pip install huggingface_hub
huggingface-cli download younghan-meta/Parakeet-TDT-ExecuTorch-CUDA-Windows-Quantized --local-dir parakeet_cuda_windows_quantizedRun
Windows (PowerShell):
.\cmake-out\examples\models\parakeet\Release\parakeet_runner.exe `
--model_path parakeet_cuda_windows_quantized\model.pte `
--data_path parakeet_cuda_windows_quantized\aoti_cuda_blob.ptd `
--audio_path C:\path\to\audio.wav `
--tokenizer_path parakeet_cuda_windows_quantized\tokenizer.modelOptional flags:
--timestamps segment— timestamp granularity:none|token|word|segment|all(default:segment)
Export Command
pip install "nemo_toolkit[asr]"
python examples/models/parakeet/export_parakeet_tdt.py \
--backend cuda-windows \
--dtype bf16 \
--qlinear_encoder 4w \
--qlinear_encoder_group_size 32 \
--qlinear 4w \
--qlinear_group_size 32 \
--qembedding 8w \
--output-dir ./parakeet_cuda_windows_quantizedCross-compilation requires x86_64-w64-mingw32-g++ on PATH and WINDOWS_CUDA_HOME pointing to the extracted Windows CUDA package. See the Parakeet README for detailed setup steps.
Note: tile_packed_to_4d packing format is not supported for cuda-windows (the cross-compilation path keeps tensors on CPU during export and aten::_convert_weight_to_int4pack only has a CUDA kernel). The quantization uses the generic IntxWeightOnlyConfig path instead.