Nanite-Labs/nanites-isles-speech-tegan

nanites-isles-speech-{hamish,maisie,seamus,tegan,rhys}

Anonymized, dialect+gender-pooled British Isles English text-to-speech voices — Scottish, Irish, and Welsh English as crowdsourced in the OpenSLR 83 dataset (Google, 2018–19). Built on CosyVoice2-0.5B (Apache-2.0, from FunAudioLLM), fine-tuned separately per pool.

Personas

Pooled voices ship under dialect-appropriate codenames so each identity is a persona, never a single OpenSLR speaker:

\* Scots lines are deliberately written in Scots orthography (e.g. "dinna fash", "ged your tea"); whisper normalizes them to standard spelling, so WER is inflated by orthography, not speech errors. The Irish/Welsh lines use standard English spelling and their WERs are reflective.

In code/config the voices keep the functional names isles-speech-scottish-female etc. (engine + dialect + gender are metadata); the codenames are the display/persona names.

Hugging Face repos are persona-named (a persona gets its own repo; a future voice gets a new codename, not a gender suffix):

• —Nanite-Labs/nanites-isles-speech-hamish (Scottish male)
• —Nanite-Labs/nanites-isles-speech-maisie (Scottish female)
• —Nanite-Labs/nanites-isles-speech-seamus (Irish male)
• —Nanite-Labs/nanites-isles-speech-tegan (Welsh female)
• —Nanite-Labs/nanites-isles-speech-rhys (Welsh male)

Why this approach

• —Engine. Same rationale as the sister smoky suite: modern line is CosyVoice 2.0 (FunAudioLLM), Apache-2.0. XTTS is retired in this project (dormant, CPML license, poor clustering/cloning control).
• —Identity. A regional anon voice must not replicate any single OpenSLR speaker. CosyVoice2 SFT collapses each dialect+gender pool into one pooled speaker ID (<|{pool}_pool|>), so no single speaker is recoverable at synthesis time.
• —Anonymization. Generation feeds the pooled campplus embedding (average of the pool's members) — no reference audio from any individual. Max cosine of the pooled embedding to the closest member is reported per voice above (0.78–0.91; smoky reference ~0.89–0.91). Seamus's pool is thin (3 speakers — the smallest OpenSLR set; the Irish female pool does not exist in OpenSLR 83, so Irish female is not offered): its anonymization margin and training-data size are inherently smaller than the 6-speaker pools.

Stable persona (zero-shot anchoring)

SFT mode conditions identity only through the pooled embedding, and the base speech flow can leak timbre from the token sequence — some short/locked-phrase sentences deterministically land in a different timbre region (diagnosed on Hamish: one sentence regen 3× identical but cosine 0.58 from his anchor). Fix, adopted for all five voices: zero-shot anchoring. Each voice dir carries anchor.wav + anchor.txt (a pooled-blend clip chosen by the listener, so the anchor is itself anonymized); cosyvoice_infer clones that clip's timbre for every new sentence (cosine 0.83–0.94 vs the anchor across novel lines). Inference is therefore persona-consistent per codename.

Data & license

• —OpenSLR 83 — "Crowdsourced high-quality UK and Ireland English Dialect speech data set" (Google, 2018–19), CC BY-SA 4.0. Citation: Demirsahin et al., "Open-source Multi-speaker Corpora of the English Accents in the British Isles", LREC 2020.
• —CC BY-SA carries over to these voices: derivatives (incl. synthesized redistribution) inherit share-alike. This differs from smoky (public-domain Hall corpus).
• —OpenSLR 83 offers 6 dialects × male/female; this release covers Scottish (f+m), Irish (male), Welsh (f+m). The Irish female set is not offered because OpenSLR 83 only ships irish_english_male.zip.
• —Audio: 48 kHz mono studio recordings → pooled 24 kHz clean sets with light spectral-gate denoise, energy-VAD trim, SNR ≥ 8 dB filter, per-pool caps (6 speakers / ~60 min / 10 min per speaker).

Training

• —Base: FunAudioLLM/CosyVoice2-0.5B (llm + flow + hift).
• —Data: pooled chunk sets → kaldi-style dirs (10% cv) → campplus embeddings + speech tokens → parquet → cosyvoice/bin/train.py (--model llm, 1×GPU, bf16 AMP, constant-lr 1e-5, 60 epochs). One speaker ID per pool.
• —Guarding (8 GB GPU): bitsandbytes AdamW8bit; non-LLM params frozen; bf16; gradient checkpointing; KEEP_CKPT=12 in-run pruning of checkpoints (multi-speaker SFT writes ~1 GB/epoch → ~60 GB if unpruned).
• —Outputs: llm_avg.pt per voice (best-8 by val loss via average_model.py --val_best); final voice dirs assembled with base flow/hift + baked spk2info.pt + anchor + samples.

Evaluation

• —WER: whisper medium.en on novel pool dialect lines (see table; target <12%; Scots lines orthography-inflated). Tools: experts/isles/eval_wer.py, eval_texts.py.
• —Anonymization: check_anon.py — pooled embedding max cosine to a member, per voice (see table).
• —Persona consistency: anchor-mode synthesis lands at cosine 0.83–0.94 vs the anchor across novel lines (vs 0.58 drift pre-fix).

Report to User

• —[x] Whisper WER < 12% on held-out lines (per voice; see table)
• —[x] Anonymization check (max cos 0.78–0.91)
• —[x] Persona-consistent anchored synthesis (0.83–0.94 vs anchor)
• —[x] All five voices signed off by listening; training data pruned
• —[ ] Publish zips via publish_cosyvoice.py --voice ... (or all)