laion/emotional-roleplay-finetuning-dataset

Artificial Voice Roleplay Dataset

67,491 fully-synthetic speech clips (~184 hours) pairing expressive role-play / character voice-direction captions with generated audio, across German, English, Spanish, and French (German-dominant). Rich in exaggerated fantasy/creature voices (orc, goblin, troll, ogre, zombie, dragon, demon, witch, banshee, imp, fairy, gnome, robot, murloc, harpy, skeleton, ghost, vampire …) and high-arousal emotional delivery (rage, fear, grief, menace).

Every audio file is machine-generated by a fine-tuned MOSS-TTS-Local v1.5 model (laion/moss-1.5-roleplay-finetune). Because the audio is synthetic — not sourced from any recorded corpus — and the captions are automatic factual descriptions, this dataset is released under CC-BY-4.0.

What's new in this release. Every clip now carries three VoiceCLAP-commercial attributes (genuineness, vocal_burst_blend, voiceclap_commercial_embedding) and two derived labels (is_human, archetype). The dataset is additionally shipped as WebDataset tar shards under `webdataset/` (plus a non-human-only subset under `webdataset/non_human/`). The original parquet files under data/ are unchanged. See the sections below and the one-page [analysis report](#analysis-report).

Highlights

• —Judge-verified labels: every voice_description was written by gemini-3.5-flash listening to the generated clip, so the label matches the audio that actually exists (TTS drift is corrected, not propagated). Clips where the judge found the wrong text or unintelligible audio are dropped.
• —Creature coverage: ~40% of clips (27,126) are non-human character voices across 20+ archetypes, including many female-voiced creatures (female orcs, dragons, banshees, harpies …).
• —Expressive range: neutral studio voices → intense emotional acting → shouting battle-cries and whispered menace.
• —Per-clip acoustic scores: VoiceCLAP-commercial genuineness (0-6) and vocal-burst-blend (0-10) for filtering / reward modeling, plus a 768-d embedding per clip.

Schema

Both the parquet (data/) and the WebDataset (webdataset/) releases carry the same fields. In WebDataset each sample is three members — <id>.mp3, <id>.json, <id>.emb.npy — with the self-explanatory JSON keys shown below.

For training a voice-direction TTS, use voice_description (+ text) — it describes the audio. Filter realized coverage with realized_gender; req_* capture intent (useful, but the model does not always comply — see limitations).

Added attributes — VoiceCLAP-commercial

Every clip was decoded, resampled to 16 kHz, and embedded once with VoiceCLAP-commercial (`laion/voiceclap-commercial`) as a 768-d, L2-normalized vector (the same convention used by the reward/generation pipelines). That single embedding feeds two lightweight commercial regression heads:

Observed ranges on this dataset: genuineness ∈ [0.00, 5.70], vocalburstblend ∈ [0.00, 10.00]. Human clips score notably higher on genuineness (mean ≈ 0.96) than synthetic creature voices (≈ 0.4-0.66), as expected.

Derived labels — is_human and archetype

These two columns are DERIVED in post-processing (there is no explicit archetype column in the source data, and no Gemini archetype field). They are computed from source + id + instruction + voice_description:

• —`is_human` — True for real / human-voiced provenance (source ∈ {ears, emolia, emotional_va, gemini, fill_human}); False for the character/creature generation tracks (source ∈ {fill_creature, character}).
• —`archetype` — "human" for human sources; otherwise the specific non-human archetype extracted by a case-insensitive word-boundary keyword map (fairy, orc, zombie, goblin, dragon, robot, demon, ghost, vampire, witch, troll, ogre, imp, gnome, murloc, banshee, harpy, skeleton, elf, angel, alien, spongebob, pikachu, cookie-monster, …). fill_*/force_* ids encode the archetype directly (e.g. fill_goblin-…); it falls back to "other-character" when no keyword matches.

Note: the base dataset's voice_description / adherence_score are gemini-3.5-flash judgments made while listening to each clip (a per-clip confirmation step); the is_human/archetype labels here are a deterministic derivation on top of that, not a separate Gemini pass.

Archetype statistics

40,365 human clips vs 27,126 non-human (40.2% non-human).

WebDataset release

Under `webdataset/`: 2 tar shards (~2 GB each, 3.13 GB total, 67,491 samples). Each sample:

<id>.mp3       # mono MP3 @ 24 kHz
<id>.json      # all metadata (self-explanatory keys) + is_human, archetype, genuineness, vocal_burst_blend
<id>.emb.npy   # float32[768] L2-normalized VoiceCLAP-commercial embedding

import webdataset as wds
base = "https://huggingface.co/datasets/laion/emotional-roleplay-finetuning-dataset/resolve/main/webdataset"
url = base + "/erp-{00000..00001}.tar"          # full set (2 shards)
# url = base + "/non_human/erp-nonhuman-{00000..00000}.tar"  # non-human subset only
ds = wds.WebDataset(url).decode()
for s in ds:
    audio = s["mp3"]; meta = s["json"]; emb = s["emb.npy"]  # 768-d
    print(meta["archetype"], meta["genuineness"], meta["vocal_burst_blend"])

Non-human subset

Under `webdataset/non_human/`: the character/creature clips only (is_human == false) — 1 shard, 0.84 GB, 27,126 samples — for training or evaluating non-human / creature voices in isolation.

Composition (67,491 clips)

How it was made

1. 1.Prompts were drawn from DramaBox-style voice-direction datasets, then augmented with a targeted grid (creature × gender × volume × emotion) and a forceful best-of-N pass aimed at hard out-of-distribution combos (female creatures, shouting).
2. 2.Audio was synthesized instruction-conditioned by the MOSS creatures_p9 fine-tune at temperature 1.7, top_p 0.8 on 2× RTX 3090. For OOD attributes the model was sampled N× and filtered locally with a VoiceCLAP gender/intensity scorer before captioning.
3. 3.Labeling + QA: each surviving clip was captioned by gemini-3.5-flash (audio in), which also verified the spoken text and intelligibility; failures were dropped.
4. 4.Post-processing (this release): every clip was scored with VoiceCLAP-commercial (genuineness, vocal_burst_blend, 768-d embedding) and given derived is_human/archetype labels, then repackaged as WebDataset.

Usage

from datasets import load_dataset
ds = load_dataset("laion/emotional-roleplay-finetuning-dataset", split="train")
ex = ds[0]
print(ex["voice_description"], "||", ex["text"], "||", ex["language"], "||", ex["realized_gender"])
ex["audio"]  # {'array': ..., 'sampling_rate': 24000}

Analysis report

A single self-contained analysis page (metadata overview, first-5 samples with audio players, archetype statistics, and per-category top-10 grids by normalized genuineness+vocal_burst_blend and by genuineness alone) is generated from this release. It is served on demand via a temporary tunnel; regenerate locally from the parquet + score files if the link has expired.

Limitations

• —Male/calm default: the generating model resists female and loud/shouting deliveries for out-of-distribution character voices; the set is still male-skewed and req_* overstate female/loud versus what was realized — trust voice_description / realized_gender.
• —Loudness labels: "shouting" is described via intensity words rather than a literal volume tag; MOSS normalizes amplitude.
• —VoiceCLAP scores are model estimates (commercial regression heads), not human ratings; use as soft signals. Synthetic creature voices score low on genuineness by construction.
• —Synthetic speech: expect occasional TTS artifacts despite intelligibility filtering.
• —voice_description is a single automatic judge's perception; not human-verified.
• —Language skews German/English; Spanish/French are small.

Licensing

CC-BY-4.0. All audio is newly synthesized (no recorded audio from any source corpus is redistributed); captions/transcripts are short automatic text. The generating model is an Apache-2.0 base model fine-tune.