sugiv/fish-speech-1.5-tanglish

Fish-Speech 1.5 · Tanglish LoRA (full_v1)

Zero-shot text-to-speech fine-tuned to produce natural Tanglish — casual code-mixed Tamil transliterated into the Latin alphabet — in the voice of a Chennai/Coimbatore speaker.

Fine-tune of `fishaudio/fish-speech-1.5` using LoRA r=8, alpha=16 (0.96 % trainable params) on 24 hours of Tanglish audio from the companion dataset sugiv/tanglish-audio-v1.

Training metrics (verified)

Eval (Phase 2, 36 WAVs on L4 24 GB SECURE)

• —15 Tanglish sentences on default speaker (karthik_kumar_solo)
• —6 Tanglish sentences across the other 5 speakers (cross-speaker A/B)
• —15 identical sentences on stock fish-speech-1.5 (A/B baseline)
• —Reference clip: karthik_kumar_solo/Ow1q3_xSJWw_000000.wav (2.52 s)
• —Reference text: "innaiku vandhadhukku mannichitten. problem!"
• —Inference: temperature=0.7, top_p=0.7, repetition_penalty=1.5, seed=42, bf16
• —Throughput: ~20 tokens/s on L4, 163 s wall for 36 samples

Full eval writeup with human-listener protocol in `eval/PHASE_2_EVAL.md`. Raw WAVs under eval/{in_distribution,novel,hard,cross_speaker,ab_base}/.

Native-speaker verdict (2026-07-14): intelligible + usable Tanglish; prosody is somewhat robotic and there are minor audio-quality artifacts — expected for 0.96 % trainable LoRA on 24 h of data with only -20 % valloss over the base model. See `PHASE2_EVAL.md` for the detailed rubric.

Contents

What is missing (and why)

The VQGAN vocoder (firefly-gan-vq-fsq-8x1024-21hz-generator.pth) is not bundled here — it is unchanged from the base model. Download it once from `fishaudio/fish-speech-1.5` and cache it locally.

Usage

import torch
from huggingface_hub import snapshot_download, hf_hub_download
from fish_speech.inference_engine import TTSInferenceEngine
from fish_speech.models.text2semantic.inference import launch_thread_safe_queue
from fish_speech.models.vqgan.inference import load_model as load_decoder_model
from fish_speech.utils.schema import ServeReferenceAudio, ServeTTSRequest

# 1. Tanglish LLM half (this repo)
llm_dir = snapshot_download("sugiv/fish-speech-1.5-tanglish", token="hf_...")

# 2. VQGAN vocoder (upstream, unchanged)
vq_ckpt = hf_hub_download(
    "fishaudio/fish-speech-1.5",
    "firefly-gan-vq-fsq-8x1024-21hz-generator.pth",
)

llama_queue = launch_thread_safe_queue(
    checkpoint_path=llm_dir, device="cuda", precision=torch.bfloat16, compile=False,
)
decoder = load_decoder_model(
    config_name="firefly_gan_vq", checkpoint_path=vq_ckpt, device="cuda",
)
engine = TTSInferenceEngine(
    llama_queue=llama_queue, decoder_model=decoder,
    precision=torch.bfloat16, compile=False,
)

ref_audio_bytes = open("ref.wav", "rb").read()
for r in engine.inference(ServeTTSRequest(
    text="machi, nalaikku Chennai la enna weather?",
    references=[ServeReferenceAudio(audio=ref_audio_bytes, text="innaiku vandhadhukku mannichitten. problem!")],
)):
    if r.code == "final":
        sr, audio_np = r.audio
        break

Known limitations

• —Prosody is somewhat wooden — LoRA r=8 alpha=16 has too few trainable params to fully overcome the base model's English + Mandarin prosody prior. Backlog item: retrain at r=32 alpha=64 with same corpus.
• —Requires an in-domain reference clip (2-4 s) — the model is a zero-shot TTS conditioned on a speaker prompt. Use one of the 6 speakers from sugiv/tanglish-audio-v1 for best results.
• —Not compatible with `transformers.from_pretrained` — fish-speech uses a bespoke Lightning-checkpoint format. Load via fish_speech.inference_engine.

License

CC-BY-NC-SA-4.0 (inherited from fishaudio/fish-speech-1.5). Non-commercial only.

Citation

@misc{tanglish_fish_speech_2026,
  title={Fish-Speech 1.5 Tanglish LoRA},
  author={sugiv},
  year={2026},
  url={https://huggingface.co/sugiv/fish-speech-1.5-tanglish}
}

Related

• —Companion audio dataset: sugiv/tanglish-audio-v1
• —Companion Qwen text agent: sugiv/qwen3-8b-tanglish
• —Base model: fishaudio/fish-speech-1.5