syvai/plapre-nano-v2
plapre-nano-v2
Danish multi-task text-to-speech. A 335M LlamaForCausalLM that autoregressively predicts Kanade 25 Hz audio tokens from Danish BPE text and decodes them to 24 kHz speech. A 128-d speaker embedding — extracted from any reference clip with the Kanade encoder — is projected and prepended to the sequence, so every generation is voiced.
What it can do
Two tasks:
Four controls that stack onto generate:
Combining them
Controls are composable — any subset stacks in one prompt, in this block order:
[SPK] [pronunciation] [voice-reference] [context] <text> BPE [pace] <audio> …Voice-reference is the only control that swaps the speaker embedding (to the reference clip's); the others keep the target voice. Edit composes with pronunciation only. All combinations are trained, not emergent.
How to use it
The easiest path is the plapre library, which wraps every task and combination:
from plapre import Plapre
tts = Plapre("syvai/plapre-nano-v2")
# plain TTS
tts.speak("Hej, hvordan har du det?", output="out.wav", split_sentences=True)
# clone a voice from any clip
tts.clone("Denne sætning har stemmen aldrig sagt.", reference_wav="voice.wav")
# cloned voice + set pace + pinned pronunciation, in one call
tts.clone(
"Mette Frederiksen mødte Volodymyr Zelenskyj i København.",
reference_wav="voice.wav",
durations=[14, 22, 12, 24, 4, 18], # one frame count per word
pronunciations=[("Zelenskyj", "zelenskyj_ref.wav")],
)
# continue a conversation with matching prosody
tts.continue_context("Og det er derfor, vi handler nu.",
prev_text="Situationen har ændret sig markant.",
prev_wav="previous_line.wav", speaker_wav="voice.wav")
# edit a recording: replace words in place
tts.edit("… ny formulering her …", mask_start=40, mask_end=55,
original_wav="clip.wav")Manual inference (transformers)
import numpy as np, torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
CKPT = "syvai/plapre-nano-v2"
tok = AutoTokenizer.from_pretrained(CKPT)
m = AutoModelForCausalLM.from_pretrained(CKPT, torch_dtype=torch.float32).eval()
spj = torch.nn.Linear(128, m.config.hidden_size)
spj.load_state_dict(torch.load(hf_hub_download(CKPT, "speaker_proj.pt"), map_location="cpu"))
spj.eval()
g = tok.convert_tokens_to_ids
STOPS = [g("</audio>"), tok.eos_token_id] # stop on BOTH terminators
pre = [g("<text>")] + tok.encode(text, add_special_tokens=False) + [g("<audio>")]
pe = m.get_input_embeddings()(torch.tensor(pre))
spk = spj(torch.tensor(np.asarray(speaker_embedding), dtype=torch.float32)).unsqueeze(0)
inp = torch.cat([spk, pe], 0).unsqueeze(0)
out = m.generate(inputs_embeds=inp,
attention_mask=torch.ones(inp.shape[:2], dtype=torch.long),
max_new_tokens=500, do_sample=True, temperature=0.7,
top_p=0.95, top_k=50, eos_token_id=STOPS,
pad_token_id=tok.eos_token_id)[0].tolist()
audio_base = g("<audio_0>")
content = []
for t in out:
if audio_base <= t < audio_base + 12800:
content.append(t - audio_base)
elif content:
break
# decode `content` with kanade_tokenizer (frothywater/kanade-25hz-clean) -> 24 kHz wavControl-block prompt formats (reference audio caps, <dur_j> ids, edit masking/splicing) are implemented in `plapre/tasks.py` — pure token-layout builders you can read or reuse directly.
Inference requirements
- Stop on both terminators: generated audio ends
</audio>then<eos>— pass both ids as stop tokens (the library does this for you). - Run in fp32 (the training precision). Lower precision measurably increases end-of-utterance artifacts.
- Use this repo's tokenizer (vocab 21224).
- End sentences with terminal punctuation — append a "." if your text ends on a comma or nothing; the stop signal is strongest on sentence-final text.
- Split long text into sentences and generate them as a batch; join with ~250 ms of silence. Speaking pace follows the voice: reference clips from calm speakers yield calm narration.
- If serving with vLLM, use
vllm>=0.15,<0.16withenable_prompt_embeds=True— newer stacks measurably degrade generation quality on identical weights. - Text normalization: collapse whitespace and convert digits to Danish words (
num2words,lang="da") before encoding.
Model details
Limitations
- Voice cloning transfers the broad character of a voice (timbre, pace, register), not a fine-grained identity — the speaker embedding is one vector per recording.
- Trained on a single speech domain; voices far outside it clone less faithfully.
- Sampling-based generation can occasionally mis-speak; for user-facing products, verify outputs with an ASR pass and resample on mismatch (a reference implementation ships with the plapre library's demo server).