John9007/MOSS-TTS-Local-GGUF

MOSS-TTS-Local-Transformer GGUF

First known GGUF quantization of [MOSS-TTS-Local-Transformer](https://huggingface.co/OpenMOSS-Team/MOSS-TTS-Local-Transformer) (1.7B parameters).

The OpenMOSS team has only published GGUFs for their 8B Delay model. This repo provides quantized GGUFs for the lightweight 1.7B Local variant, suitable for edge/on-device TTS deployment.

Available Quantizations

Architecture

MOSS-TTS-Local is a two-component TTS system:

Text → [Qwen3 backbone (1.7B, 28 layers)] → audio tokens → [ONNX decoder] → 24kHz PCM
                    ↕
         [Local transformer (4 layers)]
                    ↕
         [Bridge MLPs (33 codebooks)]

• —Backbone: Qwen3 architecture (hidden_size=2048, 28 layers, 16 attention heads, GQA 8 KV heads)
• —Audio components: 32 codebook embeddings, 33 LM heads, 33 layer norms, 4-layer local transformer, 33+1 bridge MLPs
• —Total tensors: 555 (after skipping duplicate text embedding)
• —Audio decoder: MOSS-Audio-Tokenizer-ONNX (encoder 1.5MB + decoder 14MB)

How This Was Converted

The standard convert_hf_to_gguf.py (both upstream llama.cpp and the OpenMOSS fork) fails on the Local variant because 213 audio-specific tensors lack GGUF name mappings. We added:

1. 1.19 new `MODEL_TENSOR` entries in gguf-py/gguf/constants.py for local transformer layers, bridge MLPs, and audio layer norms
2. 2.~70 lines of tensor mapping in convert_hf_to_gguf.py handling:
3. 3.Embedding offset: model.embedding_list.0 = text (skip), 1-32 = audio codebooks 0-31
4. 4.Local transformer with layer indexing
5. 5.Indexed bridge MLPs (33 per projection type)
6. 6.Quantizer must be built from the OpenMOSS fork (upstream doesn't know moss-tts-delay architecture)

Full conversion guide: see the accompanying PR on OpenMOSS/llama.cpp.

Inference

Current Status

The GGUF files contain all 555 tensors correctly. However, the llama-moss-tts C++ binary currently only supports the 8B Delay model's tensor layout (375 tensors). The Local variant's extra tensors need C++ loading + inference code in moss-tts-delay.cpp.

PyTorch inference works — audio quality verified with the original safetensors producing clean English speech at 24kHz.

Audio Quality Verification

Tested with PyTorch bf16 on RTX 3090:

• —"Hello, I am LUI, your personal AI assistant. How can I help you today?" → 5.8s clean audio
• —"Your stress has been elevated for the past hour..." → 5.6s clean audio
• —"Playing rain sounds. Notifications muted..." → 7.0s clean audio

Future: On-Device Mobile TTS

Target: Android phone with 8GB RAM using hot-swap loading:

1. 1.Unload chat LLM (Qwen 2.5 1.5B, 940MB)
2. 2.Load MOSS-TTS-Local Q3KM (1.4GB) via llama.cpp
3. 3.Generate audio tokens
4. 4.Decode with ONNX audio decoder (15.5MB)
5. 5.Play audio
6. 6.Unload TTS, reload chat LLM

Reproduction

# Prerequisites
pip install torch transformers==5.0.0 numpy sentencepiece gguf
git clone https://github.com/OpenMOSS/llama.cpp.git moss-fork

# Apply tensor mapping patches (see PR)
# Then:

# Download model
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('OpenMOSS-Team/MOSS-TTS-Local-Transformer', local_dir='moss-tts-local')"

# Convert
python3 moss-fork/convert_hf_to_gguf.py moss-tts-local/ --outfile moss-tts-local-bf16.gguf --outtype bf16

# Quantize (must use fork's quantizer)
cd moss-fork && mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release && cmake --build . --target llama-quantize -j$(nproc)
./bin/llama-quantize ../../moss-tts-local-bf16.gguf ../../moss-tts-local-Q3_K_M.gguf Q3_K_M

Credits

• —Model: OpenMOSS-Team/MOSS-TTS-Local-Transformer (Apache 2.0)
• —Conversion infrastructure: OpenMOSS/llama.cpp fork
• —GGUF conversion + quantization: obirije
• —Context: Built for LUI, an open-source Android AI assistant with on-device TTS