Nanite-Labs/nanites-smoky-speech-earl

nanites-smoky-speech-male / nanites-smoky-speech-female

[image]

Anonymized, gender-pooled Smoky Mountain English text-to-speech voices — the 1930s regional dialect of the Great Smoky Mountains (NC + TN) as recorded by Joseph Sargent Hall in 1939. Built on CosyVoice2-0.5B (Apache-2.0, from FunAudioLLM), fine-tuned separately per gender.

Personas

The two pooled voices ship under Appalachian codenames so the identity is a persona, never a single Hall speaker:

• —Earl (male) — the featured voice from this experiment. Sample picks: sample/earl_m0.wav, sample/earl_featured.wav.
• —Ethel (female) — sample pick: sample/ethel_f0.wav.

In code/config the voices keep the functional names smoky-speech-male / smoky-speech-female (engine + pool gender are metadata); "Earl" and "Ethel" are the display/persona names.

Hugging Face repos are persona-named (a persona gets its own repo; a future male or female voice gets a new codename, not a gender suffix):

• —Nanite-Labs/nanites-smoky-speech-earl
• —Nanite-Labs/nanites-smoky-speech-ethel

Why this approach (and why not XTTS)

• —Engine. Coqui is no longer an active lab; XTTS / XTTS-v2 is a poor choice for new projects (dormant maintenance, CPML license, no native batch-inference, brittle Py3.13 stack). The respected modern line is CosyVoice 2.0 / 3.0 (FunAudioLLM = the Qwen / Alibaba team), Apache-2.0. Do not pick XTTS for future speech work in this project.
• —Identity. A regional anon voice must not replicate any single Hall speaker. XTTS/F5 do speaker cloning at inference (reference clip = the person). CosyVoice2 SFT collapses 15 speakers per gender into one pooled speaker ID, so nothing original is exposed at synthesis time.
• —Anonymization. Each gender = one spk embedding (average of the pool's campplus embeddings). Generation feeds the pooled embedding — no reference audio from any individual.

Status

• —✅ v0.1 XTTS zero-shot (historic, kept only as documentation).
• —❌ v0.2 XTTS-v2 finetune: training completed but output was incoherent (probes: all checkpoints → near-silent noise; GPT latents off-manifold). Path abandoned.
• —✅ v0.4 (this model): CosyVoice2-0.5B LLM SFT per gender on 15-speaker pooled datasets (male ~42.9 min / 284 clips; female ~31.9 min / 210 clips; mainland data from USC public-domain 1939 recordings, denoised, VAD-trimmed, SNR-filtered).
• —XTTS fully retired 2026-09-07: XTTS training runs, venv, speech zip, code, and MODEL_CARD_xtts_speech.md removed; publish paths now target the CosyVoice2 voices (publish_cosyvoice.py).

Training

• —Base: FunAudioLLM/CosyVoice2-0.5B (llm.pt + flow.pt + hift.pt)
• —Data: pooled per-gender chunk sets → kaldi-style dirs (wav.scp/text/utt2spk) → campplus embeddings + speech tokens via the CosyVoice tools → parquet → cosyvoice/bin/train.py (--model llm, 1×GPU, bf16 AMP, constant-lr 1e-5, use_spk_embedding: True). Single speaker ID per gender (male_pool / female_pool).
• —Guarding (8 GB GPU): bitsandbytes AdamW8bit; non-LLM params frozen; whole model cast bf16; gradient checkpointing (COSYVOICE_GC=1 + config.use_gradient_checkpointing); DDP wrap skipped at world_size 1 (avoids checkpoint-DDP conflict); GradScaler disabled for bf16. All in the local /tmp/opencode/CosyVoice repo clone (not upstream).
• —Outputs: llm_avg.pt per gender (best-8 by val loss via average_model.py --val_best); final voice dirs assembled with base flow/hift.

Evaluation

• —Male voice (smoky-speech-male, persona "Earl"): whisper medium.en WER 1.5% (68 words) on novel dialect lines. Female voice (smoky-speech-female, persona "Ethel"): WER 4.1% (74 words). Target <12%.
• —Anonymization: pooled campplus embedding has max cosine 0.888 (male) / 0.909 (female) to the closest of the 15 pool members — a blended mean identity; generation matches no single speaker. Generation uses SFT mode (inference_sft with the pooled male_pool/female_pool embedding) — no reference clip, so no individual is exposed. (A zero-shot path with the 20 s pooled ref was trialed but produces garbled audio; SFT mode is the supported one.)
• —Fresh samples: data/cosyvoice/voices/deliverable_samples/.

Report to User

• —[x] Whisper WER < 12% on held-out dialect lines (male 1.5% / female 4.1%)
• —[x] Anonymization check (pool embedding max cos 0.89/0.91 to a member)
• —[x] experts/smoky/data/cosyvoice/voices/smoky-speech-{male,female}/
• —[ ] Publish zips via publish.py --kind speech