CoolFace
Modelpublic

lemuriandezapada/MOSS-TTS-Local-Transformer-v1.5-awq-int4

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
0likes86downloads
Model Card

MOSS-TTS-Local-Transformer-v1.5 AWQ INT4

4-bit AWQ quantization of OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5 (the 4B streaming member of the MOSS-TTS v1.5 family, 48 kHz): 8.5 GB → 3.7 GB, quality parity on ASR round-trip evaluation.

Method

  • —The Qwen3-4B backbone (transformer.*) was extracted as a standard Qwen3ForCausalLM (the checkpoint's text_lm_head.weight standing in as lm_head for calibration), quantized with AutoAWQ 0.2.9 (GEMM, 4-bit, group size 128, zero-point, pileval text calibration), and re-injected into the original layout with the transformer. prefix restored.
  • —Everything audio-specific stays untouched in the original dtype: the 12 per-codebook audio_embeddings and audio_lm_heads, the 1-layer GPT-style local_transformer (depth transformer), local_text_lm_head, text_lm_head, embed_tokens, and all norms — declared in quantization_config.modules_to_not_convert.
  • —The codec (OpenMOSS-Team/MOSS-Audio-Tokenizer-v2) is a separate repo and unaffected.
  • —torch_dtype: float16 (AWQ GEMM kernels; vLLM's awq_marlin path on Hopper takes it from here).

Validation (dual-ASR round-trip CER, worst of two judges; same texts/references as the fp16 A/B)

caseAWQ int4fp16 original
English0.000, 0.0000.000, 0.000
Japanesemedian 0.091 (5 takes, 0.061–0.273)median 0.061 (5 takes, 0.061–2.30, incl. one runaway take)
code-switched JA/EN0.46, 0.490.57, 0.73

Speaker-embedding similarity to the reference voice: +0.82–0.87 (AWQ) vs +0.82–0.87 (fp16).

Serving

Loads in vllm-omni through the registered MossTTSLocalModel architecture: the talker builds its backbone from vLLM's native Qwen3Model, so the AWQ weights go through the exact same loader path as any stock AWQ Qwen3 checkpoint — no patches. Two stages (talker + codec); real streaming via stream_format=audio (the streamed WAV is 48 kHz stereo — downmix or handle both channels client-side).

Weight footprint drops ~8 GB → ~2.8 GB. To see that saving on nvidia-smi, pin the KV pool instead of letting a gpu_memory_utilization fraction size it (vLLM fills whatever the fraction allows): a per-stage engine_extras: {kv_cache_memory_bytes: 2684354560} (4 seqs × 4096 tokens at bf16) lands the talker stage at ~7.6 GB — ~15.9 GB total with the codec stage, vs ~28.5 GB for the bf16 original.

Upstream references