laion/moss-1.5-roleplay-finetune

MOSS-TTS-Local v1.5 — Role-Play & Creature-Voice Fine-Tune

A full-parameter fine-tune of `OpenMOSS-Team/MOSS-TTS-Local-Transformer-v1.5` (4.55 B params, 12-codebook RVQ @ 48 kHz) for detailed voice-direction following — tone, mood, emotion, and exaggerated character / creature voices (Zombie, Gollum-style, Orc, Goblin, Dragon, Troll, …) — in English and German (with French/Spanish coverage), using the DramaBox prompt style.

You tell the model what to say (text) and how to say it (instruction), and it performs it.

Two checkpoints in this repo - root (./) → `creatures_p9` — recommended. Best for role-play, character & creature voices, emotional direction. Creature performance sits at the ground-truth ceiling. - `roleplay_p8_clean/` → `roleplay_p8` — the clean-audio model before the creature pass; slightly stronger neutral studio-English narration, no creature emphasis.

Results

Reference-free proxy metrics (higher = better) vs. ground-truth toplines. vc_prompt (voiceclap-small) measures voice↔direction match (reliable for English); vc_text (voiceclap-transcriptions) measures speech↔transcript match (the reliable signal for German/French/Spanish).

Creature directions are followed as well as the real creature recordings; English role-play is above the topline; German intelligibility is near-topline; runaway generations are eliminated.

Usage

import torch, torchaudio
from transformers import AutoModel, AutoProcessor

repo = "laion/moss-1.5-roleplay-finetune"          # creatures_p9 (root)
# repo, subfolder = "laion/moss-1.5-roleplay-finetune", "roleplay_p8_clean"  # alternative

processor = AutoProcessor.from_pretrained(repo, trust_remote_code=True,
                                          codec_weight_dtype="fp32", codec_compute_dtype="bf16")
processor.audio_tokenizer = processor.audio_tokenizer.to("cuda")
model = AutoModel.from_pretrained(repo, trust_remote_code=True,
                                  dtype=torch.bfloat16, attn_implementation="sdpa").to("cuda").eval()

# Split the prompt at the quote: "how to say it" -> instruction, "what to say" -> text
conversation = [[processor.build_user_message(
    instruction="You are a wretched, obsessive creature, your voice a strangled, hissing whisper "
                "torn between fawning desperation and sudden menace.",
    text="My precious... yes, yes, we wants it, we needs it.",
    language="English",
)]]
batch = processor(conversation, mode="generation")
out = model.generate(input_ids=batch["input_ids"].to("cuda"),
                     attention_mask=batch["attention_mask"].to("cuda"),
                     max_new_tokens=1000, do_sample=True,
                     audio_temperature=1.2, audio_top_p=1.0, audio_top_k=25)
audio = processor.decode(out)[0].audio_codes_list[0]     # stereo [2, samples]
torchaudio.save("out.wav", audio.cpu().float(), processor.model_config.sampling_rate)

Prompting tip: always keep the spoken words in text and the performance direction in instruction — that matches how the model was trained and gives the highest fidelity. Vivid second-person directions ("You are a massive ancient dragon, your voice a subterranean rumble…") work well for creatures.

Training

Full fine-tune (no LoRA) on 2× RTX 3090 via DeepSpeed ZeRO-3 with the optimizer offloaded to CPU.

• —Peak LR 2e-5, constant schedule (no warmup/decay); AdamW β=(0.9, 0.95), wd 0.1, bf16.
• —Global batch 32 (per-device 4 × grad-accum 4 × 2 GPUs), gradient checkpointing, loss weighting text:audio = 1:32 across the 12 RVQ heads.
• —creatures_p9 = phase 8 (2 epochs from base on a clean-audio mix) then 1 further epoch continuing from it, adding creature + multilingual-emotional data while keeping the phase-8 data as an anchor against forgetting.

Training data

(An earlier multilingual dataset, TTS-AGI/annotated-audio-tts-training, was used in exploratory phases but excluded from the final clean-audio model — in-the-wild audio was found to dilute studio prompt-adherence.)

Full pipeline, experiment log, and ablations: https://github.com/laion (see the project repo).

License & intended use

Released under CC-BY-NC-4.0 (non-commercial): the base model is Apache-2.0, but part of the training data (emotional-voice-acting-subset) is CC-BY-NC-4.0, so this derivative inherits a non-commercial restriction. For research, prototyping, and creative/role-play use.

Limitations

• —vc_prompt is only a reliable metric for English audio; German/French/Spanish are judged by vc_text plus human listening.
• —French/Spanish role-play were validated on multilingual sets, not a dedicated role-play benchmark.
• —Neutral studio-English narration is marginally weaker than a narration-specialised checkpoint (use roleplay_p8_clean/ if that is your priority).