Scicom-intl/Malaysian-Emilia-Nonverbal-Tags

Malaysian-Emilia Non-verbal Tags

Inline non-verbal event tags (laughter, cough, sigh, sneeze, burping, humming) mined from Scicom-intl/Malaysian-Emilia podcast segments, for expressive-TTS training. Tags follow the <|sfx:family|> + onomatopoeia convention (e.g. <|sfx:laughter|>Haha), placed inline at the word boundary nearest the event onset.

Example:

Dato' cakap ni macam ada pengalaman dengan orang berusia. Ada ke tau? <|sfx:laughter|>Hahaha Tak adalah...

Pipeline

audio segment (mp3, VAD-trimmed podcast speech)
   │  PANNs Cnn14_DecisionLevelMax framewise SED (~100 fps AudioSet posteriors)
   ▼  low thresholds (high recall): family = max over member classes
candidate events (onset/offset/peak)
   │  CLAP (laion/clap-htsat-unfused) zero-shot verification:
   ▼  class prompts vs speech/music/noise negatives, keep fam-prob >= 0.55
verified events
   │  faster-whisper large-v3 word timestamps
   ▼  insert tag at nearest word gap; events overlapping speech >50% kept in
      metadata but NOT tagged (speech-laugh is a different phenomenon)
tagged transcript

Calibration note: AudioSet laughter posteriors are heavily suppressed on speech-adjacent laughter (median clip-max 0.007 on Expresso laughing clips), so SED thresholds are scaled to ~0.3x of naive values and precision comes from the CLAP gate (median verified CLAP confidence >= 0.95 for all classes).

Files

• —data/tagged-*.parquet — one row per audio segment with >= 1 verified event: file (segment basename), orig_text (source transcript), whisper_text, tagged_text (whisper transcript with inline tags), language, events (JSON list: family, onset, offset, peak, clapfam, clapneg, placed), n_placed
• —events/events-*.jsonl — all SED candidate events pre-verification (for re-thresholding)
• —qa_crops/ — per-class event crops (wav, 48 kHz) for auditing, filename carries CLAP confidence

Text columns

Each row carries two renderings of the same verified events:

• —tagged_text — Higgs-TTS style: <|sfx:family|> + onomatopoeia, only for discrete events (speech overlap <= 50%)
• —nv_text — Emilia-NV / NVSpeech style: bare [Label] word-level tokens for ALL events at their occurrence position, including vocalizations overlapping speech (e.g. laughing while talking)

Stats (all 10 shards, ~1.28M segments ≈ 3,000 h)

8,702 tagged transcripts, 8,985 verified events (ms 6,594 / en 2,093).

Note: burping is dominated by mouth/plosive sounds that CLAP accepts — audit qa_crops/ before training on that class. sigh/crying recall is limited by weak AudioSet classes.