Modelpublic

mesmertech/Breeze-TTS-2-int4-hqq-g64

sourceHugging Faceapache-2.0updated 23d agoView on Hugging Face

1likes143downloads

Model Card

Breeze-TTS-2 int4 (HQQ g64, pre-packed)

Weight-only int4 derivative of BreezeBlue/Breeze-TTS-2 (revision c1c8ca18b70b, Apache-2.0). This is not a drop-in `transformers` checkpoint -- it needs the breeze-tts-runpod loader (int4_tinygemm.py + load_packed.py).

What was quantised

280 bias-free nn.Linears: the 84 in depth_decoder.model.layers[0..11] and the 196 in backbone_model.layers[0..27] (self_attn.{q,k,v,o}_proj, mlp.{gate,up,down}_proj).

Never quantised: the T5Gemma2 text encoder, lm_head, the codebooks head, every embedding, the two projectors, codec_model, and the Qwen3-TTS audio_tokenizer/. Those tensors are here verbatim, in their original dtype and under their original names.

3.25 GiB of bf16 decoder weights become 0.91 GiB of int4 + bf16 scales.

Recipe

item	value
method	HQQ proximal solver (hqq 0.2.8.post1), `nbits=4, axis=1, channel_wise, optimize=True, round_zero=False`, no calibration data
group size	64, along K
solver arithmetic	float32 on the build host. HQQ picks float16 only when `device.type == "cuda"` (`hqq/core/optimize.py:231`); this checkpoint was built on an Apple M1 Pro, so HQQ's own non-CUDA float32 branch ran. The resulting per-Linear weight errors match the CUDA/float16 gate run to four decimals (0.0889 depth / 0.0879 backbone).
dequant	`w = (q - 8) * scale + zero`, `q` in `[0, 15]` (torch `int4mm.cu` convention)
`<m>.qweight`	`uint8 [N, K/2]`, `byte j = (q[:, 2j] << 4) \	q[:, 2j+1]`
`<m>.scales_and_zeros`	`bfloat16 [K/64, N, 2]`, `[..., 0] = scale`, `[..., 1] = zero`
kernel	`torch._weight_int4pack_mm(x_bf16, packed, 64, scales_and_zeros)`, sm_80+
weight error (Frobenius, per Linear, with the bf16-rounded scales the kernel really sees)	depth mean 0.0889 / max 0.0982; backbone mean 0.0879 / max 0.0932

Why "pre-packed"

qweight is stored in the portable nibble layout -- exactly the tensor that torch._convert_weight_to_int4pack takes as input -- not in torch's tensor-core-tiled output layout, which is architecture- and version-specific (innerKTiles semantics changed at torch 2.5) and has no CPU equivalent. So a loader does:

python

packed = torch._convert_weight_to_int4pack(qweight.cuda(), 2)   # milliseconds
mod = Int4Linear(packed, scales_and_zeros.cuda(), K, N, 64)

instead of re-running the quantiser at every boot (0.6 s for RTN g128, 3.4 s for HQQ g64 on an RTX 4090). Shipping HQQ pre-packed is the only reason HQQ is practical here: it has the lower weight error of the two validated variants but was too slow to redo per process.

Quality

The recipe (not this artefact) passed a 320-clip distributional gate on an RTX 4090: 5 texts x {design, clone} x 8 seeds x 4 variants, scored with UTMOS22, DNSMOS, Whisper WER, resemblyzer speaker cosine, pyin F0 and words/s. No int4 variant was distinguishable from bf16 beyond the bf16 seed-to-seed noise floor; HQQ g64 had the lowest clone WER. Speed on a 4090: 47.7 -> 33.5 ms per codec frame (design), 41.3 -> 31.5 (clone); VRAM after warmup 7.53 -> 5.20 GiB.

Files

file	bytes
`model-int4.safetensors`	4,461,414,090
`audio_tokenizer/model.safetensors`	682,293,092
`tokenizer.json`	33,386,945
`tokenizer_config.json`	1,157,960
`model.safetensors.index.json`	123,791
`quant_stats.json`	54,108
`int4_manifest.json`	19,875
`config.json`	10,161
`README.md`	4,976
`audio_tokenizer/config.json`	2,336
`special_tokens_map.json`	886
`generation_config.json`	251
`audio_tokenizer/preprocessor_config.json`	234
`audio_tokenizer/configuration.json`	76

model.safetensors.index.json is regenerated and maps every tensor to the single model-int4.safetensors. config.json is the upstream one, verbatim -- it still says text_encoder_config.preferred_attn_implementation = "flash_attention_2", which breeze-tts-runpod/model_setup.py:ensure_attn_patch rewrites to sdpa at boot.

Loading

<m>.weight does not exist for the 280 quantised modules, so transformers will report them as missing and report .qweight / .scales_and_zeros as unexpected. See load_packed.py in breeze-tts-runpod/optimization-research/J-packed-checkpoint/ for the two construction routes and the load_packed_int4(model, packed_dir, device) reference implementation. The swap must happen after load_runtime and before Engine.warmup(), which is the CUDA-graph capture.

Licence

Apache-2.0, inherited from BreezeBlue/Breeze-TTS-2. This repository contains only a derived (quantised) form of those weights plus the upstream tokenizer/config files verbatim.