malaiwah/qwen3-tts-customvoice-ab-clips

qwen3-tts: full 5-way cloning comparison + cross-row diagnostic

Generated 2026-04-14 on RTX 4080 SUPER.

Directories

original/ CustomVoice.generatecustomvoice(speaker=X) -> the ground truth voice clone/ Base.generatevoiceclone(refaudio=original.wav, reftext=...) -> full ICL clone via Base's own speaker encoder transplant/ Base.generatevoiceclone(voicecloneprompt=[row]) xvectoronlymode=True, refcode=None, reftext=None -> row lifted from CustomVoice.codecembedding[spkid] hybrid/ Base.generatevoiceclone(voicecloneprompt=[row + refcode]) xvectoronlymode=False, iclmode=True -> row for identity, refcode from original.wav for prosody crossrow/ (diagnostic) refcode from speaker A + row from speaker B -> identity lives in the row, prosody in refcode

Cosine similarity between the two speaker-conditioning paths

For each speaker, we compute: enc = Base.speakerencoder(originalwav) # ECAPA-TDNN output, 2048-d row = CustomVoice.talker.codecembedding[spkid] # 2048-d

speaker cos(enc, row) L2(enc - row) |enc| |row| aiden +0.974 3.90 16.95 15.57 dylan +0.965 4.50 16.80 15.05 eric +0.954 5.57 17.64 15.35 onoanna +0.965 4.83 16.98 14.54 ryan +0.934 6.60 17.15 13.89 serena +0.964 4.97 17.25 14.67 sohee +0.965 4.58 17.42 15.52 unclefu +0.952 5.63 17.57 15.02 vivian +0.965 5.18 17.74 15.09

The encoder output has slightly larger magnitude (~17 vs ~14-16), but the directions are nearly identical. That's why original/, clone/, and transplant/ are acoustically indistinguishable — the 2048-d conditioning signal lands in the same place either way.

Suggested A/B listening

Per speaker, four-way compare:

afplay original/ryanEnglish.wav afplay clone/ryanEnglish.wav afplay transplant/ryanEnglish.wav afplay hybrid/ryanEnglish.wav

Then the cross-row sanity check:

afplay crossrow/refcode-vivianrow-ryan.wav afplay crossrow/refcode-ryan__row-vivian.wav

In cross_row, the prosody tokens come from one speaker's audio and the identity row from another. If identity really lives in the row, these files should speak in speaker B's voice with speaker A's cadence.

Why this matters

1. 1.Runtime cost of transplant vs ICL clone: steady-state identical (both ~2.1 s / ~8 s utterance on 4080 SUPER).
2. 2.Registration cost: 8.3 s for ICL clone vs 0 ms for row transplant.
3. 3.Sidecar size per voice: 13 KB (ICL) vs 4 KB (row only).
4. 4.Shipping the 9 CustomVoice voices on top of -Base is possible with 36 KB of sidecar data — no extra checkpoint needed.
5. 5.speaker_encoder can be dropped at load time (~24 MB VRAM) if you only ever use pre-computed rows and never re-register from raw audio.

Run timing (RTX 4080 SUPER, bf16)

phase mean wall notes CustomVoice built-in 2.2 s direct token lookup Base clone (refaudio path) 2.17 s ECAPA-TDNN + speech tokenizer Base transplant (xvectoronly) 2.16 s no preprocessing at all Base hybrid (row + refcode) 2.19 s same as ICL clone, swapped x-vector

Research status

To the best of our knowledge (web search done 2026-04-14), the exact "lift a row from a CustomVoice-style sibling checkpoint and feed it as refspkembedding to the Base inference path" technique is not documented publicly anywhere — not in the Qwen3-TTS community nor for any other TTS family (CosyVoice, XTTS-v2, F5-TTS, VALL-E, StyleTTS2, OpenVoice, Spark-TTS, Fish Speech, Kokoro, Bark). Prior work exists on speaker adapters (Hsieh et al. 2022, Kwon et al. 2025) but stays within one checkpoint. Worth writing up.