CoolFace
Datasetpublic

laion/tts-scaling-ladder-de-en

Balanced Expressive Speech: German–English Scaling Views A nested set of ten cumulative training views for reproducible expressive TTS and audio-model studies. Eight base tiers were designed to be approximately 50/50 German/English and, within each language, 50/50 real/synthetic; each cell reserves half its volume for coverage of 40 EmoNet emotion categories and half for 57 VoiceNet dimensions × 10 value bins. Two later, non-overlapping extensions yield cumulative views of 49… See the full description on the dataset page: https://huggingface.co/datasets/laion/tts-scaling-ladder-de-en.

sourceHugging Faceotherupdated 2d agoView on Hugging Face
1likes327downloads
Dataset Card

Balanced Expressive Speech: German–English Scaling Views

A nested set of ten cumulative training views for reproducible expressive TTS and audio-model studies. Eight base tiers were designed to be approximately 50/50 German/English and, within each language, 50/50 real/synthetic; each cell reserves half its volume for coverage of 40 EmoNet emotion categories and half for 57 VoiceNet dimensions × 10 value bins. Two later, non-overlapping extensions yield cumulative views of 49,569.2 and 98,920.1 measured hours. The larger view falls short of the nominal 100,000-hour build target; it is not a 100,000-hour measurement. Every smaller training view is contained in the next larger one. The 80.1-hour holdout is separate.

The release is intended to support future from-scratch TTS training, scaling comparisons, and exploration of broad-to-selective curricula. The from-scratch Humaneness Voice follow-up was not completed at the time of the manuscript submission; the existing Humaneness Voice baseline is a fine-tune of MOSS, not a model trained on this full ladder. Source-specific audio rights apply throughout, including EuroSpeech's parliament-specific terms. The full ladder is not uniformly commercially permissive even though its metadata are publicly documented.

Every element ships audio (MP3, byte-identical to its source repository), MOSS-Audio-Tokenizer-v2 codes (uint16 [T, 12], 12.5 fps) and a JSON record with the procedural voice-acting prompt in the `LAION-AI/procedural-voice-captions` format (GENERAL line + SCRIPT with per-sentence delivery cues, [pause X.Xs] markers and inline (Vocal Burst) labels), plus the raw scores it was built from, the language, the origin, the source, the source uid, the source licence and the duration.

Human-readable cumulative viewExisting loader configMeasured training hoursClips
Broadstage10098,920.134,112,433
Expandedstage5049,569.217,135,901
Coretier720,001.56,666,998
Tier 6tier610,000.73,425,099
Tier 5tier55,000.41,764,990
Tier 4tier42,500.1912,789
Tier 3tier31,000.0382,768
Tier 2tier2500.1197,662
Tier 1tier1249.9100,858
Tier 0tier0100.041,771
Holdout (not training)heldout80.125,639

How to read this table. Durations are measured cumulative views, not ten additive datasets. stage50 contains all of Core plus 29,567.7 extension hours; stage100 contains all of Expanded plus 49,350.9 further extension hours. The existing configuration names are retained for code compatibility; Broad, Expanded and Core are explanatory display names. An illustrative, untested curriculum would train for one epoch on Broad, then one epoch on each next-smaller view, down to Tier 0. Because sets nest, high-ranked clips are deliberately revisited. The schedule has not been validated as better than alternatives and does not describe training of the current MOSS-based Humaneness Voice checkpoint. Filter per-record licence fields before any use that requires a narrower rights set; doing so changes measured hours and balance.

Held-out split. 20 h per cell (10 h per pipeline) drawn with exactly the same rules as the tiers — the next elements behind the tier-7 boundary of every bucket/category in master-rank order, same normalisation, same quotas — and disjoint from all tiers (tier = 8, verified by uid: zero overlap). It lives under data/*/*/heldout/ and index/*/*/heldout/, loader config heldout. Use it for the scaling-law loss; never train on it.

Hours are summed over dur_s. Tier N ⊂ Tier N+1 strictly — by construction, not by check: the whole release is stored once (the tier-7 stock plus held-out, 6,692,637 elements, 20,081.6 h, 2,694 shards, 1.04 TB of tars) with a column tier = the smallest tier index that contains the element, and the tierN loader configs simply filter tier <= N.

Two assumptions made by the executor (the specification left them open)

  1. 1.50/50 split between the two pipelines. Inside every cell (language × origin) half of the volume is filled by pipeline A (uniform over 40 EmoNet categories) and half by pipeline B (uniform over 57 VoiceNet dimensions × 10 deciles). The specification fixed the strata but not the share; halves were the coordinator's assumption.
  2. 2.One stock, one `tier` column. The release is stored exactly once (the tier-7 stock plus the held-out split) and every element carries tier = the smallest tier that contains it. Tier K is therefore defined as tier <= K (and, on disk, as the tar directories tier0 … tierK); nesting is guaranteed by construction rather than by shipping eight overlapping copies.

The remaining executor's choices (pool definition, decile edges, deduplication order, bucket shortfall handling, language verdicts) are marked [A] in the rules below.

Loading — one example per tier

Every tierN config is the union of the tar/parquet directories tier0 … tierN; heldout is the evaluation split.

python
from datasets import load_dataset
R = "laion/tts-scaling-ladder-de-en"
t0 = load_dataset(R, "tier0")      #    100 h  (50 DE + 50 EN; 25 h real + 25 h synth per language)
t1 = load_dataset(R, "tier1")      #    250 h
t2 = load_dataset(R, "tier2")      #    500 h
t3 = load_dataset(R, "tier3")      #  1,000 h  compute-optimal anchor
t4 = load_dataset(R, "tier4")      #  2,500 h
t5 = load_dataset(R, "tier5")      #  5,000 h
t6 = load_dataset(R, "tier6")      # 10,000 h
t7 = load_dataset(R, "tier7")      # 20,000 h  everything
ev = load_dataset(R, "heldout")    #     80 h  evaluation, disjoint from every tier
de = load_dataset(R, "de")         # one language, all tiers: filter de["train"]["tier"] <= N yourself

Each row is one element (metadata, scores, prompt); audio and MOSS codes are in the tar with the same basename:

Audio and MOSS codes live in WebDataset tars next to the index, one tar per index parquet, same basename:

python
import io, json, tarfile, numpy as np
from huggingface_hub import hf_hub_download
row = idx["train"][0]                                       # any index row
p = hf_hub_download("laion/tts-scaling-ladder-de-en", f"data/{row['lang']}/{row['origin']}/tier{row['tier']}/{row['shard']}.tar", repo_type="dataset")
with tarfile.open(p) as tf:
    mp3  = tf.extractfile(f"{row['uid']}.mp3").read()               # bytes, decode with soundfile/torchaudio
    moss = np.load(io.BytesIO(tf.extractfile(f"{row['uid']}.moss.npy").read()))   # uint16 [T, 12]
    rec  = json.load(tf.extractfile(f"{row['uid']}.json"))          # prompt, words, sentences, bursts, scores
print(rec["prompt"])

Tar members per element: <uid>.mp3, <uid>.moss.npy, <uid>.json. Every tar of tier block tierN holds only elements whose smallest tier is N, so tier K = all tars under tier0 … tierK. For streaming with webdataset:

python
import webdataset as wds
K = 3
urls = [f"https://huggingface.co/datasets/laion/tts-scaling-ladder-de-en/resolve/main/data/{lang}/{origin}/tier{n}/{shard}.tar"
        for (lang, origin, n, shard) in shards_of_tier(K)]      # from the index: columns lang, origin, tier, shard
ds = wds.WebDataset(urls).decode().to_tuple("mp3", "moss.npy", "json")

Decoding note. The voice-profile sources (vprof_original, vprof_repair) are 48 kHz MP3 with two identical channels; decode with soundfile/torchaudio and take mean(axis=channels) — never reshape(-1) (that yields a half-speed signal of twice the length). All other sources are mono. audio_sr and audio_channels are in the index.

Selection rules (verbatim from the specification, with the executor's assumptions marked [A])

Licence filter (before any scoring)

Only sources with documented provenance and release terms entered this published pool; this is not a claim of uniform commercial reuse rights. Real: emolia (DE+EN), mls (DE; the corpus has no English MLS), eurospeech (DE+EN, explicitly released by the project lead), kartoffelphon (DE), commonvoice (DE and EN subsets only). Synthetic: laion-voice-profiles-annotated (origin=original, origin=repair, vc_sidon — 500 LAION voice profiles, DE/EN in halves), laion/voice-acting-edge-top3 (formerly laion/dramabox-edge-top3-voice), laion/voice-acting-reinterpretations-top3 (formerly laion/dramabox-reinterpretations-top3) (MOSS takes). Excluded: podcast, snippets, evasnippets (research repo, non-commercial), everything Mediathek, everything without a documented licence, laion/voice-acting-burst-audio (no score columns), laion-emotional-trajectory-t80 (chains concatenated from vc_sidon clips — duplicates — and 93 % language-mixed), laion/moss-voice-identity-repairs (identical material to origin=repair, which carries the MOSS codes and scores).

Scoring — all metrics min-max normalised per language × origin cell

S_gen = Norm(genuineness_0_6), S_vb = Norm(blend_0_10), S_emo,e = Norm(emo_e) for the 40 EmoNet dimensions, S_vn,d = Norm(vn_d_reg) for the 57 VoiceNet dimensions. R_quality = S_gen + S_vb; R_emotion(e) = 3·S_emo,e + R_quality. [A] min/max are exact over the whole cell pool (all sources of the cell together).

Stratification

Per cell (language × origin) the cell volume is split in halves between two pipelines ([A], coordinator's assumption).

  • —Pipeline A — EmoNet, 40 categories. For category e: candidates sorted by R_emotion(e); the intensity pool [A] is defined as nested bands: band k (k = 0…7) = the top prefix whose hours reach m · q_A(k) with m = 3 (the pool covers the quota threefold), qA(k) = pipeline-A volume of tier k / 40. **Master rank** in A = (band ascending, then `Rquality descending). Tier 0 is therefore the best-quality third of the most extreme band; every larger tier adds less extreme emotion and again ranks strictly by quality inside. Quota per category = 1/40 of the pipeline-A volume. If a category's pool is exhausted by deduplication, the ranking continues below band 7 in R_emotion(e) order (band = 8`, counted in the report).
  • —Pipeline B — VoiceNet, 57 dimensions × 10 deciles = 570 buckets. For each dimension the value range [min, max] of the cell is cut into 10 equal-width intervals ([A]: equal width, not quantiles); inside each bucket the master rank is strictly R_quality. Quota per bucket = 1/570 of the pipeline-B volume, so extreme values (very low voice, very young age) are not disadvantaged. [A] A bucket with fewer available hours than its quota takes everything it has; the shortfall is redistributed proportionally to the other deciles of the same dimension (then, if needed, across all buckets of the pipeline).
  • —Deduplication. An element counts once. [A] Fixed order: pipeline A first, then B; inside A the 40 categories in alphabetical order (all pools are equal-sized); inside B the 570 buckets in ascending order of available candidate hours (scarcest first, which protects the extreme deciles), ties by (dimension, decile). An element already taken is skipped by later buckets. Elements that would have fallen into more than one window are counted (n_candidacies).
  • —Nesting. Per bucket/category the deduplicated list is walked in master-rank order, hours accumulated, and each element receives tier = min{N : cum_h ≤ q(N) + dur/2} (rounding to the nearest quota); elements beyond q(7) are dropped. [A] Bucket rounding noise is then balanced at the tier boundaries by moving boundary elements (last of tier n ↔ first of tier n+1) until every (cell, pipeline, tier) is within 0.25 % of its target — the prefix property is preserved. Stored once with tier, rank_in_bucket, bucket_id, pipeline, emotion or vn_dim/decile, plus band (A) and n_candidacies.
  • —[A] Language = the corpus lang verdict (upstream label + ASR LID, lang_src == agreed on 99.6 %); for the DramaBox sources fastText lid.176 over the ASR transcript. Duplicate uids inside a source (Emolia ships some clips in two upstream archives) are dropped before selection, first occurrence kept. No duration filter beyond dur_s > 0.

Licences per source

source (`source`)HF repositoryoriginlicence of the audionote
emolialaion/laion-tts-annotated-v1 (emolia)realCC-BY-4.0 (laion/Emolia)
commonvoicelaion/laion-tts-annotated-v1 (commonvoice)realCC-0 (Mozilla Common Voice)DE + EN subsets only
eurospeechlaion/laion-tts-annotated-v1 (eurospeech)realother — parliament terms vary; the upstream card makes the user responsible for complianceincluded on the project lead's explicit release
kartoffelphonlaion/laion-tts-annotated-v1 (kartoffelphon)realCC-BY-4.0 (~51 % LibriVox public domain)
mlslaion/laion-tts-annotated-v1 (mls)realCC-BY-4.0 (facebook/multilingual_librispeech)German only
vprof_originallaion/laion-voice-profiles-annotated (origin=original)synthCC-BY-4.0; generator laion/moss-tts-local-transformer-4.55b-voice-acting-v2 (Apache-2.0)500 synthetic voice profiles; 124 mediathek_* voices are model-reinterpreted speakers, not broadcast audio
vprof_repairlaion/laion-voice-profiles-annotated (origin=repair)synthCC-BY-4.0re-dos of takes that failed an identity check; both versions are distinct audio and both are in the pool
vc_sidonlaion/laion-voice-profiles-annotated (vc_sidon)synthCC-BY-4.0; Chatterbox VC (MIT) + SIDONvoice-conversion twin of the same takes as `vprof_original` (same script and performance, different timbre); one winning candidate per take
dramabox_edgelaion/voice-acting-edge-top3 (formerly laion/dramabox-edge-top3-voice)synthCC-BY-4.0 (synthetic material from LAION's own generation; annotations Apache-2.0)266 hand-written edge cases × 2,000 prompts, rendered with DramaBox TTS (Resemble AI weights), top-3 of 20 seeds per group; language per clip from the ASR transcript
dramabox_reintlaion/voice-acting-reinterpretations-top3 (formerly laion/dramabox-reinterpretations-top3) (takes)synthCC-BY-4.0 (synthetic material from LAION's own generation)MOSS-voice-acting-v2 reinterpretations (top-3 of 64 candidates) of DramaBox acting prompts; the unannotated DramaBox originals of that repo are not included

The license column carries this per element. If your use case cannot accept other, filter license to CC-BY-4.0 / CC-0 (this removes eurospeech; the tiers stay nested but no longer exactly balanced).

Balance (hours, from the shipped index)

tierde_real Ade_real Ben_real Aen_real Bde_synth Ade_synth Ben_synth Aen_synth BDEENrealsynth
012.512.512.512.512.512.512.512.550.050.050.050.0
131.231.231.331.231.331.331.231.3125.0125.0124.9125.0
262.562.562.562.562.562.562.562.5250.0250.0250.0250.0
3125.0125.0125.0125.0125.0125.0125.0125.0500.0500.0500.0500.0
4312.5312.5312.5312.5312.5312.5312.5312.61,250.01,250.11,250.01,250.1
5625.0625.0625.0625.1625.0625.0625.0625.22,500.02,500.32,500.12,500.2
61,250.01,250.11,250.01,250.21,250.01,250.11,250.01,250.35,000.35,000.55,000.35,000.4
72,500.02,500.32,500.02,500.32,500.02,500.32,500.02,500.610,000.610,000.910,000.610,000.9
held-out10.010.010.010.010.010.010.010.040.140.140.040.1

Targets per (cell, pipeline), tiers 0–7: 12.5 / 31.25 / 62.5 / 125 / 312.5 / 625 / 1,250 / 2,500 h. Largest deviation over the 64 tier values: 0.204 %; over the eight held-out values: 0.238 %. All are within 0.25 %.

Hours per source (tier 7 plus held-out)

sourceDE realEN realDE synthEN synth
commonvoice170.0 (131,515)69.6 (57,578)——
dramabox_edge——156.3 (19,747)2,547.5 (310,985)
dramabox_reint——52.5 (12,805)61.4 (15,617)
emolia2,091.4 (669,330)4,915.0 (1,505,156)——
eurospeech608.1 (140,828)32.8 (7,916)——
kartoffelphon2,116.6 (611,467)2.9 (923)——
mls34.1 (7,889)———
vc_sidon——2,050.7 (709,047)690.4 (394,290)
vprof_original——2,218.8 (747,642)1,338.1 (651,627)
vprof_repair——542.0 (351,342)383.3 (346,933)

The prompt (what rec["prompt"] / index prompt contains)

Produced offline from the stored annotations with the captioner of `LAION-AI/procedural-voice-captions` at main (PR #4: 17-class x2 head and energy-based pause correction), tags template, deterministic seed = stable_hash(uid):

GENERAL: very warm, very soft-onset, …, middle-aged, low-register, very slow, pining, helplessness, semi-genuine, with-bursts
SCRIPT:
(very flat, very smooth, no interest, very fatigue, genuine) Huh. (Chuckle)
(very flat, very warm, helplessness, concern, semi-genuine) [pause 0.4s] What have we here?
  • —GENERAL — top-5 VoiceNet dimensions by |z|·reliability, top-3 emotions, genuineness, burst blend, always Age/Gender/Register/Tempo, z-scored against the bundled in-domain baseline.
  • —SCRIPT — one line per sentence: (delivery cue) sentence text. Sentences are cut at sentence-final punctuation of the aligned words. [A] The cue of every sentence is derived from the clip's global scores (the corpora carry no per-sentence scores; the record stores the sentence timings so per-sentence re-scoring can be dropped in later). Wording rotates deterministically per sentence.
  • —`[pause X.Xs]` — a gap ≥ 0.30 s between consecutive words or sentences, measured from the stored word timestamps after the energy-based word-end correction (each word's end is trimmed back to where its speech actually stops, computed from the decoded audio; end_span keeps the raw value). Word timestamps are the corpus alignments (words in the source JSON — MMS-FA for the LAION corpora, Parakeet for the DramaBox sources); Parakeet was not re-run.
  • —`(Vocal Burst)` — the kept locator/classifier-v2 spans of the source annotation, written at the word gap where they happen (BURST_LABEL_SOURCE=v2). prompt_x2 is the same prompt with the 17-class x2 head naming the spans (BURST_LABEL_SOURCE=x2, recall-tiered wording: exact class / Class? / family word / Vocal Burst) — present where the vb2_* re-annotation exists (3,689,357 of 6,692,637 elements: the five real sources, vc_sidon shards with a re-annotation, dramabox_reint, and dramabox_edge where its re-annotation had landed), null elsewhere.
  • —Totals in this release: 8,508,596 pause markers, 3,928,534 kept bursts.
  • —rec["sentences"], rec["words"], rec["bursts"] and rec["scores"] are the record augment.py needs to regenerate varied phrasings of the same content at training time (augment_script(rec, seed)).

Columns

See `COLUMNS.md`. Every index row carries the raw scores (genuineness_0_6, blend_0_10, 40 emo_*, 57 vn_*_reg and, where the source has them, 57 vn_*_bucket), lang, origin, source, uid (= source uid), hf_source, license, dur_s, the selection fields and the prompt.

Provenance and verification

Built on JUPITER (JSC) from the local copies of the source repositories; nothing was re-encoded — every MP3 is the byte-identical member of its source tar, every MOSS code array is the source's (laion-tts-annotated-v1, laion-voice-profiles-annotated) or, for the two DramaBox sources, newly computed with OpenMOSS-Team/MOSS-Audio-Tokenizer-v2 at 48 kHz, n_vq = 12, with the frame-rate canary moss_frames / (decoded_duration × 12.5) ∈ [0.97, 1.03] checked on every element (the packer refuses otherwise). Upload with upload_large_folder; every file verified byte-exact against the local copy via repo_info(files_metadata=True). Selection code, statistics and the full report: ladder.json in this repo; the working report is scaling_ladder.md in the project.

Citation

If you use this ladder, credit LAION and the upstream sources listed above.

50k and 100k extensions (2026-09-19)

This release extends the frozen 20k ladder with two incremental, non-overlapping container sets. The files are stored under logical x50/ and x100/ directories; the build system's incidental physical tier5//tier1/ source directories are not part of the public layout.

extensionTAR / Parquet pairselementsmeasured hoursapparent TAR size
x504,19010,468,90329,567.7~1.53 TB
x1006,79316,976,53249,350.9~2.89 TB
both extensions10,98327,445,43578,918.6~4.42 TB

stage50 loads the 20k tier stock plus x50 (approximately 49.6k measured hours); stage100 additionally loads x100 (approximately 98.9k measured hours). The latter is deliberately called stage100, not a claim of exactly 100,000 hours: 51 planned shards were not packed after a documented 12-hour timeout (4 de/synth, 47 en/synth), leaving the final total about 1,082 hours below its target. This is a transparent quota shortfall, not a missing manifest entry.

The original tier0–tier7 and held-out containers are unchanged. The x50/x100 records use the same WebDataset triplets and MOSS-v2 convention as the base release.

Extension release checks

Before publication, every extension Parquet was paired with its TAR and .tar.ok marker; deterministic canaries in every stage/language/origin cell verified complete {mp3, moss.npy, json} triplets, loadable uint16 [T,12] NumPy MOSS matrices, matching JSON frame counts, and a 12.5-Hz duration ratio in [0.95, 1.05]. The release audit report and canary SHA-256 values are retained with the training run.

License remains mixed per source

The repository-level identifier intentionally remains other, not CC-BY-4.0. The extension contains CC-BY-4.0 and CC-0 records and, in de/real x100, the existing eurospeech value other (parliament terms; released by project lead). The per-record license column and the source table above are authoritative. Filter to CC-BY-4.0 / CC-0 if the other terms are unsuitable for a use case.

Annotation sidecars for expressive TTS and audio-language training

Release note. This section specifies the additive annotation release that accompanies the ladder update. Publish the README and the paths below in the same commit; the annotations are not part of the older public commit 92ae1518f80088a64372138dcfd450a0290e6e66.

The completed annotation pass targets 3,682,649 unique clips, consisting of 3,425,099 clips from the base ladder and 257,550 clips from the synthetic voice-conversion sidecar. Its production directory and historical physical manifest are named S3, but the corrected ten-stage curriculum calls this full-coverage set logical S4, or tier6 + sidecar tier6. It is not the whole public tier7 stock. Every target UID has a non-empty BUD-E Whisper V1.1 caption, Timbre-Whisper output, Voice-Tagging-Whisper output, and one validated Gemma-4-E4B DramaBox-style instruction.

Public file contract

The strings are stored once in aligned Parquet containers and joined by uid; the smaller stages use compact membership Parquets instead of copying millions of captions:

text
annotations/
  s3/
    whisper/whisper.part-00000.parquet ... whisper.part-00255.parquet
    gemma/gemma.part-00000.parquet     ... gemma.part-00255.parquet
    ANNOTATION_DATASET.json
    FINAL_SUMMARY.json
  ladder-membership/
    S5.membership.parquet
    S6.membership.parquet
    S7.membership.parquet
    S8.membership.parquet
    S9.membership.parquet
    S10.membership.parquet
    FINAL_SUMMARY.json
  bude-v1/bude_caps.parquet

The two files with a matching five-digit part number contain the same UIDs in the same order. ANNOTATION_DATASET.json records row counts and SHA-256 hashes for all 256 pairs. Each membership row is simply {uid: string, annotation_part: int16}. The authoritative v2 descriptor maps the logical curriculum as follows:

logical stagephysical ladder viewannotation coverage
S1x100the 3,682,649 annotated tier6 UIDs embedded in this superset; other rows retain the procedural/legacy prompt
S2x50the same embedded tier6 UID set; other rows retain the procedural/legacy prompt
S3tier7the same embedded tier6 UID set; other rows retain the procedural/legacy prompt
S4tier6 (historical physical manifest S3, nominal 20k source)all 3,682,649 annotation UIDs; full unique-UID coverage
S5tier5full coverage via S5.membership.parquet
S6tier4full coverage via S6.membership.parquet
S7tier3full coverage via S7.membership.parquet
S8tier2full coverage via S8.membership.parquet
S9tier1full coverage via S9.membership.parquet
S10tier0full coverage via S10.membership.parquet

Only logical S5-S10 require membership Parquets. Logical S4 is exactly the annotated tier6 UID set, while logical S1-S3 are larger supersets with partial annotation coverage. The published logical filenames normalize older internal physical labels: S5 sources physical/S4, S6 sources physical/S5, S7 sources physical/S5b, S8 sources physical/S6, S9 sources physical/S5c, and S10 sources physical/S7.

The original audio and codec targets remain in the WebDataset TARs under data/<lang>/<origin>/<tier-or-extension>/<shard>.tar. Sidecars never duplicate audio. The Whisper Parquet columns record_path, record_member, and audio_member are provenance locators from the build workspace; portable loaders should resolve a clip from its ladder index fields (lang, origin, tier, shard, uid) rather than relying on those absolute build paths.

What each annotation means

fieldproducer and public modelpurpose
record_json.promptprocedural-voice-captionsDeterministic GENERAL: plus per-sentence SCRIPT: conditioning derived from measured VoiceNet, EmoNet, genuineness, burst and timing annotations. record_json.text is the verified transcript.
bude_captionlaion/BUD-E-Whisper_V1.1Free-form emotional speech caption with transcription/context, emotion, delivery, recording quality, gender/reverb/background wording. It is enrichment, not a replacement for the verified transcript.
timbre_captionlaion/timbre-whisperRaw combined output: structured timbre taxonomy followed by natural-language timbre prose.
timbre_tags_json, timbre_prosedeterministic parser over timbre_captionSeparately conditionable tags and prose. Tag order may be shuffled during training; an absent part remains empty and is never fabricated.
voice_tags_captionlaion/voice-tagging-whisperStructured quality, naturalness, fluency, style, phonation, airflow, loudness, prosody, articulation and delivery tags.
voice_tags_json, voice_tags_prosedeterministic parser over voice_tags_captionTags/prose split. Production Voice-Tagging outputs are normally tag-only, so voice_tags_prose is usually empty by design.
instruction, variantgoogle/gemma-4-E4B-it plus deterministic rendererOne of six natural English direction formats. All words to be spoken remain verbatim inside straight double quotes; directions, optional verified durations and vocal-burst instructions remain outside.
raw_generation, render_json, validation_json, passGemma audit trail and deterministic gatesRaw direction JSON, renderer audit, exact-transcript/quote/timing/identity validation, and final pass flag. The production release has zero validation failures.
bude-v1/bude_caps.parquetlaion/BUD-E-Whisper, the original V1.0 model used for the Emolia-era caption joinHistorical {uid, bude_caption, bude_len} alternative for Emolia-source clips. This is the model sometimes called “Body Whisper”, “Buddy Whisper” or “Bude Whisper” in internal notes; the canonical public name is BUD-E Whisper.

The model revisions used for this release are BUD-E-Whisper_V1.1@9a4b11214f2f37e87a05dc75a51edd472e7bba59, timbre-whisper@0eba2ef9f74ef8a37a31774dbf9fbbd40568e6fe, voice-tagging-whisper@dc223dda0fabb29a90a9d8c16b1e8ae5ff0988fe, and gemma-4-E4B-it@ee0ef6023621cff504d758262d4e04895a5af4a2. The original BUD-E V1.0 join contains 10,629,084 unique UIDs and is useful as a historical alternative; it is not silently substituted for V1.1.

Related public datasets are laion/Emolia, laion/laion-tts-annotated-v1, laion/laion-voice-profiles-annotated, laion/voice-acting-edge-top3, and laion/voice-acting-reinterpretations-top3. The synthetic curriculum sidecar does not yet have a verified public repository; do not infer or invent a Hub identifier for it.

One real, sanitized aligned example

The following text comes from one completed Emolia row. The UID and local filesystem locators are omitted here; stored strings are otherwise shown as produced. Raw model outputs are retained even when their wording is imperfect.

Verified transcript

text
Die größten Herausforderungen waren im letzten Jahr.

Original BUD-E Whisper V1.0 caption (historical Emolia join)

text
A medium-quality recording of a male speaker talking in a conversational tone.
He sounds somewhat neutral, possibly slightly interested. The audio quality is
decent, with no noticeable background noise.

BUD-E Whisper V1.1 caption

text
InIn a slightly subdued, neutral tone, an adult male voice expresses a mild
sense of Contentment and Interest, with a hint of Concentration. The delivery
is natural and spontaneous, featuring a slightly low-pitched, moderate-to-mid
pitch range, a somewhat monotonous and steady tempo, and clear articulation.
The language is likely German with a neutral accent, recorded in a quiet
environment with slight room reverberation.

Timbre-Whisper tags

text
male_baritone, slightly_soft_neutral, dark_neutral, slightly_breathy,
slight_nasal_touch, near_neutral_relaxed, slight_roughness, chest_mixed,
near_neutral_heavy, mild_wear, mostly_natural, stable

Timbre-Whisper prose

text
Pleasant middle-aged adult male voice with medium-pitched delivery, exhibiting
a slightly soft and dark timbre. There's a touch of breathiness and nasality,
coupled with a relaxed vocal production. The voice presents a slight roughness
and a chest-mixed resonance, possessing a near-neutral heavy vocal weight. Mild
wear is detectable, yet the voice remains mostly natural and stable.

Voice-Tagging-Whisper tags

text
Suitable for Work, natural speaking, fluent, conversational style, modal voice,
neutral airflow, normal loudness, slightly dynamic, precise articulation,
natural speaking

Procedural prompt

text
GENERAL: very understated, very relaxed, very smooth, very monotone, very dark,
adult, masculine, low-register, slightly slow, no fascination, no reflection,
no provoking lightly, genuine
SCRIPT:
(very understated, very relaxed, very smooth, no intrigue, no pensiveness,
no provoking lightly) Die größten Herausforderungen waren im letzten Jahr.

Gemma-4-E4B rendered instruction (`NR_TIMBRE_SENT`)

text
A very dark, low-register, smooth adult male voice speaks with a very
understated and relaxed tone. Speak with a very understated, very relaxed,
very smooth delivery. "Die größten Herausforderungen waren im letzten Jahr."

Parquet schemas and joining

The Whisper Parquets contain:

text
uid, source, mode, dur_s, record_path, record_member, audio_member, variant,
lang, record_json, bude_caption, timbre_caption, timbre_tags_json,
timbre_prose, voice_tags_caption, voice_tags_json, voice_tags_prose

The aligned Gemma Parquets contain:

text
uid, source, mode, variant, record_path, record_member, instruction,
raw_generation, render_json, validation_json, pass,
quote_punctuation_canonicalized, generated_tokens,
batch_gpu_seconds_per_output, model_path

Minimal zero-copy-style part join with PyArrow:

python
import json
import pyarrow.parquet as pq

part = 0
w = pq.read_table(f"annotations/s3/whisper/whisper.part-{part:05d}.parquet")
g = pq.read_table(f"annotations/s3/gemma/gemma.part-{part:05d}.parquet")
assert w.column("uid").to_pylist() == g.column("uid").to_pylist()

# The files are aligned, so columns can be appended without a global shuffle.
row = {name: w[name][0].as_py() for name in w.column_names}
row["instruction"] = g["instruction"][0].as_py()
record = json.loads(row["record_json"])
text = record["text"]
procedural_prompt = record["prompt"]

For S5-S10, first scan the appropriate membership Parquet and group rows by annotation_part; then read only those of the 256 aligned part pairs. This keeps both metadata I/O and the number of open files bounded.

MOSS Audio Tokenizer v2: frames, codebooks and token rate

This release stores the 12-codebook operating point of OpenMOSS-Team/MOSS-Audio-Tokenizer-v2:

  • —.moss.npy is uint16 with shape `[T, 12]`.
  • —The frame rate is 12.5 frames/s, so one row represents 80 ms.
  • —Every frame has 12 RVQ codebook indices. If a model flattens all indices, that is 12.5 × 12 = 150 discrete indices per second.
  • —It is therefore incorrect to describe these files as “12 tokens per second”. They contain 12 indices per 80-ms frame.
  • —MOSS v2 supports other operating points up to 32 quantizers, but this dataset intentionally precomputed 12. The omitted higher residual codebooks are not present and cannot be reconstructed from the [T,12] arrays.

Recommended training uses

For instruction-conditioned TTS, use record_json.text as the immutable speech target, choose either the procedural prompt or instruction as conditioning, and predict the [T,12] MOSS target (parallel codebooks, delayed codebooks, a Talker head, or a flattened 150-index/s sequence are all possible model-side choices). Never train the model to speak the unquoted English directions in a Gemma instruction.

For reference-conditioned TTS, select rows with mode == "reference"; the REF_* variants intentionally omit speaker identity/timbre wording because the reference audio supplies it. For no-reference voice design, use NR_TIMBRE_* or NR_ARCHETYPE_SENT. For audio-language or omni-model training, the MP3 can be the audio input and any of the captions/tags/instructions can be supervised text, retrieval metadata, or auxiliary targets. Keep heldout evaluation-only, retain raw annotations for audit, and treat disagreements between captioners as QA evidence rather than silently overwriting the verified transcript.

laion/tts-scaling-ladder-de-en · CoolFace