Nairod785/parakeet-redux-gguf

Parakeet Redux 0.6B — native ternary GGUF for transcribe.cpp

GGUF files of moondream/parakeet-redux for transcribe.cpp, keeping the model ternary: the encoder weights stay {−1, 0, +1} codes with one scale per 128 weights (1.75 bits/weight) and run on dedicated ternary kernels. Nothing is dequantized on disk.

Parakeet Redux is Moondream's ternary sibling of parakeet-ultra, itself a post-training of NVIDIA's parakeet-tdt-0.6b-v3: FastConformer encoder + TDT transducer decoder, 25 European languages, 16 kHz mono input, punctuated and cased output with optional token/word timestamps. Not a streaming model; does not translate.

• —Parent model: moondream/parakeet-redux @ 2bf128600aac4b16946f7ed8372e56117fe5e23b
• —Weights: 604 M ternary (the 264 encoder linear / pointwise-conv matrices) + 23 M dense
• —Languages (25): bg, hr, cs, da, nl, en, et, fi, fr, de, el, hu, it, lv, lt, mt, pl, pt, ro, ru, sk, sl, es, sv, uk — automatic language detection
• —License: CC-BY-4.0 (inherited; see Attribution)
• —Format details, kernels, and how to reproduce: QUANTIZATION.md

Files

The ternary encoder weights (604 M, 1.75 bits/weight) are bit-identical to Moondream's in all three files; they only differ in how the 23 M dense (never-ternary) parameters are stored. For reference the parent is model.safetensors 177.8 MB + tokenizer.json 1.2 MB; each GGUF is a single self-contained file including the tokenizer.

Why TQ1_F16 is 0.9 % larger than model.safetensors: it also embeds the tokenizer, and its fixed 256-weight blocks pack trits per 128-weight group (26 bytes) where Moondream packs them across whole rows. Against the parent's weights + tokenizer it is 0.17 % larger.

Requirements

TQ1_G128 is a ggml type added by transcribe.cpp (downstream patch patches/ggml/0003-tq1_g128-ternary.patch). You need a transcribe.cpp build that includes it; stock llama.cpp / ggml cannot read these files.

At load time the ternary weights are re-laid-out losslessly (same codes, same fp16 scales) into the in-memory format that is fastest on the backend — the file on disk stays 1.75 bits/weight:

TRANSCRIBE_TERNARY_RUNTIME=q4_0|q2_0|native overrides the choice; native keeps TQ1_G128 in memory (1.75 bpw) and uses the dedicated ternary kernels (CPU AVX2/NEON, CUDA, Vulkan) — smallest memory, slower.

Usage

cmake -B build && cmake --build build --config Release --target transcribe-cli
build/bin/transcribe-cli -m parakeet-redux-0.6b-TQ1_F16.gguf audio.wav
build/bin/transcribe-cli -m parakeet-redux-0.6b-TQ1_F16.gguf -l de --timestamps word audio.wav

Validation

• —Numerical parity against the reference (transformers ParakeetForTDT, fp32, with Moondream's ternary weights dequantized exactly) on samples/jfk.wav: 18/18 tensors within tolerance with the TQ1F16 file, transcript identical (for comparison, the standard Q80 of parakeet-ultra passes 16/18 of the same tolerances).
• —Kernels: packing is lossless (random codes and scales round-trip exactly); the CPU kernel matches an integer reference; mul_mat on CPU / CUDA / Vulkan matches a float reference (tests/ternary_tq1_g128_unit.cpp).
• —Accuracy:

Same recipe as parakeet-ultra's card (FLEURS French test, 676 utterances, greedy, no LM, CUDA, batch 1). The three files are indistinguishable; the gap to parakeet-ultra (4.65 %) is the model's own ternary compression, not the conversion — the C++ output matches Moondream's weights run in transformers tensor for tensor.

Speed

Measured with transcribe-bench on a 29.3 s clip (samples/german.wav), warm, mean of 3 iterations, RTX 4070 Laptop GPU (8 GB) and its laptop x86 CPU (AVX2), transcribe.cpp after the 2026-09-26 optimization round (see docs/porting/parakeet-optimization-2026-09-26.md in the repo). "×" = times faster than realtime.

Encoder / decoder split on CUDA: 40 ms / 60 ms. Accuracy of the optimized paths was re-measured (FLEURS-fr: CUDA 8.34 %, CPU 8.34 %) and is unchanged.

Differences from the parent checkpoint

• —The 6 vad_head.* tensors (Photon runtime's VAD head, ≈213 K parameters) are omitted.
• —The ternary codes are re-laid-out from Moondream's per-row base-3 packing (thrush-ternary-v2) into TQ1_G128 blocks; codes and FP16 scales are copied exactly.

Attribution

Model weights: © Moondream, released under CC-BY-4.0 as moondream/parakeet-redux, derived from NVIDIA's parakeet-tdt-0.6b-v3 (CC-BY-4.0). This repository changes only the file format (GGUF, ternary blocks re-laid-out losslessly) and, for the optional variants, the precision of the dense tensors, as described in QUANTIZATION.md.

Correction (2026-09-26): WER figures published earlier were ~1.6–1.8 pp too high. 61 of the FLEURS-fr reference transcripts in the manifest had been stored mojibake-encoded (é → é, Windows cp1252 default in the manifest builder), so correct hypotheses scored as errors. The references were repaired and every report re-scored; the numbers here are the corrected ones. The scripts now always pass an explicit encoding (scripts/ci/check_text_encoding.py).