Serveurperso/Parakeet-TDT-0.6b-v3-GGUF

Parakeet TDT 0.6B v3 GGUF

GGUF weights of NVIDIA Parakeet TDT 0.6B v3 for s2s.cpp, a C++17/GGML voice assistant. FastConformer encoder and TDT transducer decoder, 25 European languages detected on their own, punctuation and capitalization included. Runs on CPU, CUDA, Metal, Vulkan, SYCL.

Files

The server loads the best quant it finds, up to Q8_0.

Quick start

git clone https://github.com/ServeurpersoCom/s2s.cpp.git
cd s2s.cpp
git submodule update --init
./buildcuda.sh
hf download Serveurperso/Parakeet-TDT-0.6b-v3-GGUF parakeet-tdt-0.6b-v3-Q8_0.gguf --local-dir models
./build/parakeet-transcribe --model models/parakeet-tdt-0.6b-v3-Q8_0.gguf --file examples/freeman.wav

./models.sh then fetches the rest of the pipeline and ./server.sh starts the voice assistant on http://localhost:8088.

Quantization policy

The quantizer of s2s.cpp mirrors llama-quantize: every 2D weight takes the K-quant of the variant, and in the M variants the attention values and the second feed forward linear are bumped to Q6_K on the first, the last and every third layer, the two places a low bit weight hurts the transcript first. Three groups stay F32 in every variant:

The conversion folds each conformer BatchNorm into its depthwise kernel, so the graph has none.

Parity

test-parakeet holds each stage against transformers ParakeetForTDT on the example recording: mel, subsampling, first block, encoder states, projection, prediction network, joint, and the transcript.

CUDA figures, the transcripts are the same on Vulkan and CPU.

License

Upstream model: Parakeet TDT 0.6B v3 by NVIDIA, CC BY 4.0

GGUF tooling: s2s.cpp, MIT