Datasetpublic

EmpathicRobotics/voiceclap-flattened

voiceclap-flattened SNAC-tokenized, flattened + augmented build of 5 subsets of laion/voiceclap-data (CC-BY-4.0): emolia (English block), ears, expresso, voxceleb1, voxceleb2. Replaces the earlier single-subset voiceclap-emolia-flattened repo (consolidated here). Why this exists Follow-up to an ablation study (2/3/5) on <listen>/<speak> token format: full 7-tok/frame <listen> content with a SEPARATE vocab (ids shifted +1,000,000 vs <speak>'s identical SNAC codes)… See the full description on the dataset page: https://huggingface.co/datasets/EmpathicRobotics/voiceclap-flattened.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes353downloads

Dataset Card

voiceclap-flattened

SNAC-tokenized, flattened + augmented build of 5 subsets of laion/voiceclap-data (CC-BY-4.0): emolia (English block), ears, expresso, voxceleb1, voxceleb2. Replaces the earlier single-subset voiceclap-emolia-flattened repo (consolidated here).

Why this exists

Follow-up to an ablation study (2/3/5) on <listen>/<speak> token format: full 7-tok/frame <listen> content with a SEPARATE vocab (ids shifted +1,000,000 vs <speak>'s identical SNAC codes) recovered most TTS reliability while keeping most of the ASR/comprehension gain over the old truncated-3-tok/shared-vocab baseline. Huu's decision (2026-08-10): use this design + more data for the next real training run. ears/expresso/ voxceleb1/voxceleb2 add real (non-synthetic-TTS) human speech across diverse accents/recording conditions -- same domain as the ASR win was measured on (DailyTalk).

Tokenizer

Requires tokenizer-vla-qwen3-v3 (or newer) -- <listen>'s shifted <snac_{N+1000000}> ids are not atomic in earlier tokenizers.

Tasks per subset

Every task is built by deterministic field-selection + phrasing-pool templates (hash of the record id picks the pool entry) -- no LLM involved anywhere in this stage. "USER"/"ASSISTANT" below stand for "whichever side of the wrap the content lands on" -- see Format wrapping further down for the 4 literal wrap styles actually used.

`emolia_en` -- tasks A-F

Task	USER content	ASSISTANT content
A (TTS, terse)	2 sentences hash-picked from the `speaker_voice`/`style_performance`/`style_casual`/`style_authority` field-groups, followed by `[Language: English] Say: "{transcript}"`	`<speak>` block (full 7-tok/frame SNAC of the real clip)
B (ASR)	`<listen>` block + 1 of 4 pooled questions ("Transcribe this." / "What is being said here?" / "Write down exactly what is spoken." / "Provide a transcript of this audio.")	the real transcript text
C1 (speaker description)	`<listen>` block + 1 of 3 pooled questions ("Describe the speaker." / "Who does this sound like -- age, gender, register?" / "What can you tell about the speaker from their voice?")	2 sentences hash-picked from the `speaker_voice` field-group
C2 (timbre/prosody)	`<listen>` block + 1 of 3 pooled questions ("Describe the voice's timbre and prosody." / "How does this voice sound -- texture and rhythm?" / "Describe the tonal quality and pacing of the speech.")	2 sentences hash-picked from `timbre_color`/`timbre_texture`/`prosody_melody`/`prosody_speed`
C3 (recording context)	`<listen>` block + 1 of 3 pooled questions ("Describe the recording." / "What's the recording context and production quality?" / "Describe how and where this was likely recorded.")	2 sentences hash-picked from `recording_context`/`physical_production`
D (voice, no audio)	`"{transcript}" Imagine the voice.` (text-only, no `<listen>`/`<speak>`)	1 sentence hash-picked from the same field-groups as A
E (emotion)	`<listen>` block + 1 of 4 pooled questions ("What is the dominant emotional quality?" / "What emotion comes through most strongly here?" / "How would you describe the speaker's emotional state?" / "What is the main emotion conveyed in this clip?")	`"The speaker sounds {emotion phrase}."` -- `{emotion phrase}` is picked by argmax over the row's `emotion_annotation` dict, restricted to the 39/55 dims that are genuinely emotional (16 demographic/quality dims like `Age`/`Gender`/`Background_Noise`/`Recording_Quality` are excluded), mapped through a fixed phrase table (e.g. `Anger_best` -> "with noticeable anger") -- a lookup, not LLM-generated
F (TTS, rich prompt)	the row's `emotion_caption` (a denser, multi-clause LLM-authored voice+emotion description from the source dataset) followed by `Say: "{transcript}"`	same `<speak>` block as A (same audio, richer instruction)

C1/C2/C3/D/E only appear when the source row actually has non-empty text in the relevant field-group; a row missing e.g. emotion_annotation simply has no Task E record.

`ears` / `expresso` / `voxceleb1` / `voxceleb2` -- tasks A-C

These subsets use Lhotse MonoCut schema (verified 2026-08-11 across 19/20 sampled shards) and have no emotion_annotation-style numeric dims, so only 3 tasks:

Task	USER content	ASSISTANT content
A (TTS)	1 of 3 pooled lead-in templates filled from the clip's `accent`/`pitch`/`speaking_rate` attributes (e.g. "Speak with a {accent} accent, {pitch}, at a {rate}." / "Use a {accent} accent, {pitch} voice, {rate}." / "Adopt a {pitch}, {accent}-accented voice, speaking at a {rate}."), followed by `[Language: English] Say: "{transcript}"`	`<speak>` block (full 7-tok/frame SNAC)
B (ASR)	`<listen>` block + 1 of the same 4 pooled ASR questions as `emolia_en`'s Task B	the real transcript, from `supervisions[0].text` (the top-level `text`/`old_text` fields are voice-style captions, NOT the words spoken -- verified not to be confused here)
C (voice/tone description)	`<listen>` block + 1 of 4 pooled questions ("Describe the speaker's voice." / "What does this voice sound like?" / "Describe the tone and delivery of this speech." / "How would you characterize this speaker's voice?")	1 caption hash-picked from the row's `finegrained_captions` pool (falls back to `global_captions` if that pool is empty) -- these are genuine paraphrase captions written by the source dataset's authors, not generated by this project

Format wrapping

Every record (whichever task/subset) is wrapped in one of 4 literal styles, hash-picked per record id (deterministic -- the same record always gets the same style on a re-run):

1) Instruction: {USER}
   Response:
   <think>
   </think>
   {ASSISTANT}

2) User: {USER}
   Assistant:
   <think>
   </think>
   {ASSISTANT}

3) USER: {USER}
   ASSISTANT:
   <think>
   </think>
   {ASSISTANT}

4) <|im_start|>user
   {USER}<|im_end|>
   <|im_start|>assistant
   <think>
   </think>
   {ASSISTANT}<|im_end|>

Base

laion/voiceclap-data (CC-BY-4.0). Audio SNAC-tokenized (hubertsiuzdak/snac_24khz, full 3-level 7 tok/frame), <listen> ids shifted +1,000,000 vs <speak>'s.