ModelsLab/midashenglm-gen-training-latents

ModelsLab/midashenglm-gen-training-latents

Precomputed audio latents for fine-tuning `mispeech/midashenglm-gen`, paired with six-view prompts in the exact format the model was trained on.

This is not an audio dataset and not a caption dataset. Each record is the output of the model's frozen DashengTokenizer encoder — 768-dimensional latents at 25 Hz, stored float16 — next to the tagged prompt string built from the source metadata.

Why it exists

The encoder is frozen during fine-tuning, so its output for a given clip is a constant. Running it every epoch is waste, and precomputing keeps the mel front end and a 12-layer ViT out of the training step's memory budget. Building this set costs about 90 minutes, most of it a Whisper large-v3 pass over the MECAT speech categories, which carry no transcripts of their own.

Contents

Total 29,200 clips, 59 shards, float16.

Format

Each *.pt is a torch.save of a list of dicts:

{
  "id":         "libritts:1234_56_000001_000000",   # or mecat:S0A:<key>
  "prompt":     "<|caption|> ... <|asr|> ... <|speech|> ... <|sfx|> ... <|music|> ... <|env|> ...",
  "latents":    np.ndarray,   # [T, 768] float16, 25 Hz
  "seconds":    4.31,
  "kind":       "speech" | "scene",
  "category":   "",           # MECAT bucket, empty for LibriTTS
  "capability": "speech" | "sfx" | "music" | "ambience" | "mixed",
  "text":       "the transcript",   # present where one exists
}

import torch
records = torch.load("libritts-00000.pt", weights_only=False)
print(records[0]["prompt"], records[0]["latents"].shape)

Prompts are built by the repo's own build_prompt, so the six tags come out in the order the endpoint sends and absent fields come out as <|unknown|>. The tag order is load-bearing: <|asr|> before <|speech|>, measured at 14.2% mean word error against 373% reversed.

How it was built

• —Speech — LibriTTS-R, text_normalized (so the model learns "two hundred", not "200"), trimmed of leading and trailing silence and peak-normalised to 0.95 to match what the endpoint returns.
• —Scenes — MECAT-Caption, whose long / speech / music / sound / environment views map onto the model's tags directly. MECAT carries no transcripts, so <|asr|> for the speech categories comes from Whisper large-v3; clips whose transcript returns non-English or low-confidence are dropped rather than guessed at, which is why the S0A and SMA buckets are smaller than their source.
• —`<|speech|>` descriptors are measured off each waveform — median F0, words per second, spectral centroid — rather than looked up per speaker. Age, accent and texture cannot be measured honestly, so they are not invented.
• —Clips outside 1-20 seconds are dropped, following the paper's training range.

Licence and attribution

Released CC-BY-4.0, and derived from two attribution-only sources that must be credited by anyone using it:

• —LibriTTS-R (`mythicinfinity/libritts_r`), CC-BY-4.0 — Koizumi et al., LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus.
• —MECAT-Caption (`mispeech/MECAT-Caption`), CC-BY-3.0 — the public slice of ACAVCaps, itself derived from ACAV100M.

Because the model's decoder reconstructs audio from these latents at parity with the source — log-mel L1 of 0.26 against controls at 3.3-4.0, and identical word error rate — treat this as a faithful encoding of that audio rather than as a derived feature set, and honour the source licences accordingly.

Base model mispeech/midashenglm-gen is Apache 2.0.

Provenance

Built by `audio-scenegen/training/prepare_data.py`.