michael-chan-000/qwen3_voice_design_v1

Qwen3-TTS VoiceDesign — T1

A fine-tune of Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign targeting better intelligibility on natural-language voice-description prompts. The base model is steered at inference time by a free-form English instruct string describing gender, pitch, pace, and affect; T1 sharpens how faithfully the output follows that prompt while preserving the base model's decoding stack.

• —Base: Qwen3-TTS-12Hz-1.7B-VoiceDesign (frozen during training, LoRA adapter merged into this repo)
• —Method: LoRA on the Talker's attention + MLP projections, merged back into the base weights
• —Training data: TextrolSpeech (12,000 clips, stratified by pitch × speed × situational × gender)
• —Output: 24 kHz mono wav via the Qwen3 12 Hz multi-codebook codec

This repo is self-contained — it ships the merged transformer weights, the audio codec (speech_tokenizer/), the tokenizer, and all configs. You do not need to pull any other HF repo at inference time.

Results on the 16-prompt in-distribution eval

Eval format: natural-language voice-design prompts covering 8 emotions × 2 genders × 3 pitch/speed combinations. ASR: openai/whisper-base.

WER dropped from 5.11% → 1.74% (−66%), composite score +15.3%. No regression on any evaluated checkpoint.

Per-prompt wavs and raw metrics (summary.json, comparison.json) are reproducible with the same 16 prompts at temperature=0.9, top_k=50, top_p=1.0, repetition_penalty=1.05, max_new_tokens=600.

Quick start

Install the Qwen3-TTS inference package (it registers the custom Qwen3TTSForConditionalGeneration model class with transformers):

pip install qwen-tts transformers torch soundfile

Generate a clip:

from qwen_tts import Qwen3TTSModel
import soundfile as sf

wrap = Qwen3TTSModel.from_pretrained("macminix/qwen3_voice_design_t1")

wavs, sr = wrap.generate_voice_design(
    text="Come and look at this, you are not going to believe it.",
    instruct="A male speaker delivers his happy speech at a moderate pace with standard energy.",
    language="english",
    temperature=0.9, top_k=50, top_p=1.0,
    repetition_penalty=1.05, max_new_tokens=600,
)
sf.write("out.wav", wavs[0], sr)

A ready-to-run version is provided at `example_inference.py`.

The instruct prompt format

The instruct field is free-form English describing the voice. The training distribution has four axes:

• —gender — "a male/female speaker"
• —pitch — "high/medium/low pitched", "deep", "high-pitched"
• —speed — "slow/moderate/fast", "at a brisk pace"
• —situational / emotion — "happy", "angry", "sad", "whispered", "contemptuous", etc.

Example prompts:

A male speaker delivers his happy speech at a moderate pace with standard energy.
A female voice speaks slowly with a sad tone, low energy, almost whispering.
A low-pitched male narrator reads an exciting announcement at a fast pace.

How the adapter was trained

T1 was an iteration in a larger fine-tuning study that traced a regression in v1 of the same project (monotonically decreasing validation loss but audibly worse outputs at every checkpoint after step 500) to four concrete mistakes in the v1 training sequence. T1 is the first clean training under the corrected protocol.

Correctness fixes vs the naive recipe

Training hyperparameters

LoRA adapter (as trained, before merge)

The adapter before merging was 77 MB (~19.2 M fp32 parameters). That adapter has been permanently folded into the model's Talker weights in this repo; the user-facing file is a single standalone model. If you need the un-merged LoRA adapter for composition or research, it is preserved at a separate location.

Evaluation details

• —Prompts: 16 natural-language voice-description prompts covering 8 emotions (happy, angry, sad, disgusted, afraid, contempt, neutral, surprised) × 2 genders × 3 pitch/speed buckets.
• —Decoder: temperature=0.9, top_k=50, top_p=1.0, repetition_penalty=1.05, do_sample=True, max_new_tokens=600.
• —ASR: openai/whisper-base via transformers pipeline.
• —Composite score: 0.5 · (1 / max(WER, 0.05)) + 0.3 · emotion_acc + 0.2 · utmos_normalized. The T1 eval did not install the emotion classifier or UTMOS; those columns are unreported.

WER is fraction-of-tokens-wrong; lower is better. Composite is higher-is-better, unbounded (dominated by 1/WER when WER is small).

Known limitations

• —Gender × pitch confound. Prompts like "a male speaker at a high pitch" or occasionally "a male speaker at a low pitch" can produce a female-timbre voice. Root cause: the training corpus's gender distribution per pitch bucket is imbalanced — the model partially treats pitch as a proxy for gender. Not fixed in this checkpoint.
• —Emotion intensity is mixed. Some emotion prompts produce clearly stronger delivery than the base; others are roughly comparable. Emotion metrics were not logged at training time, so emotional fidelity is a subjective assessment rather than a tracked metric.
• —English only. All training and evaluation used English prompts + English text. The base model supports 10 languages; they are untouched but were not validated against this adapter's shifted CB-0 distribution.
• —Research / non-commercial only — see license.

License

• —Base model weights (Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign): Apache 2.0.
• —Training data (TextrolSpeech): CC BY-NC-SA 4.0 (research / non-commercial).

Because a substantial portion of this model's post-training behavior is shaped by TextrolSpeech, the derived model inherits the CC BY-NC-SA 4.0 constraint in practice: free to use for research, academic, and non-commercial purposes, with attribution and share-alike. Commercial deployment is not recommended without re-training on a commercially-licensed corpus.

References

• —Base model: Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
• —Inference library: `qwen-tts` on PyPI
• —Upstream LoRA training reference: QwenLM/Qwen3-TTS/finetuning/sft_12hz_lora.py
• —Training data: TextrolSpeech (Ji et al., 2023)