laion/vocal-burst-detector-x3

vocal-burst-detector-x3

### Which file do I want? `production/` — start here. The laion/voiceclap-large-v2 (3584-d, frozen) head, 5 initialisations averaged in probability space, 19 classes = 18 bursts + `no_burst`, with bs_score.py (importable scorer, unchanged API), per_class_recall.json (read it before you read any hit rate) and the full confusion matrices. `commercial/` is the same recipe on laion/voiceclap-commercial (768-d): cheaper, and — see below — the more selective rejecter. ablations/ holds the variants that explain every number here. training_extras/ carries the provenance of the new negatives. laion/vocal-burst-detector-x2 is untouched; this repository is its successor for re-scoring the laion-tts-annotated-v1 corpus.

What changed against x2, and why — three findings

x2 was used to re-classify the 63.8 M burst spans that laion/vocalburst-locator v2 + the old vocal-burst-detector-v2 had stored in laion-tts-annotated-v1. The result was that 70 % of all spans came back as `Sharp Inhale` (Mediathek: 81 %), and that the two commonest old labels — Ahem and Low Mumble, ~26 % of the old events each — were not in x2's 16-class vocabulary. The obvious reading was "x2 cannot name the two commonest sounds; add them". Three measurements turned that reading around (all in training_extras/ and the report this card summarises):

1. 1.The old detector invented its two commonest labels. In the 3,598 real utterances that carry both the old spans and a blind whole-clip gemini-3.8-flash annotation, a Gemini event overlaps an old Ahem span in 16.5 % of cases and an old Low Mumble span in 7.3 % — against 86.5 % for old Chuckle spans and 91.5 % for old Sharp Inhale spans. A level dip (≤ −10 dB) does not change this (15.2 % / 5.0 %). Where an event does overlap, it is almost never a throat-clear or a mumble (6 and 1 of ~1,250). Those spans are locator false alarms mid-speech, given the old detector's two fallback names.
2. 2.x2 never rejects them. On the old spans in the corpus, x2's P(no_burst) is 0.000 at the median and ≥ 0.5 on 0.03–0.09 % of them, whatever the level contrast. Its no_burst training rows were ≥ 0.5 s windows of speech or silence, ≥ 0.5 s clear of every annotation; a 0.2–0.5 s cut inside running speech is something it never saw as "nothing", so it picks the nearest burst class, and for a short breathy speech fragment that is Sharp Inhale. A calibration problem of the reject output, not a vocabulary problem.
3. 3.A real throat-clear is not what x2 calls `Sharp Inhale` either. On Gemini's own Ahem segments, x2 says Chuckle (real, 50 %) or Humming / Affirmative Grunt / Chuckle (DramaBox); on Low Mumble it says Humming. So the two classes are missing from the vocabulary — but adding them alone would have forced the same non-events into another class.

x3 therefore does two things: it adds `Ahem` and `Low Mumble` (Gemini labels Clears Throat and Throat-clearing folded into Ahem, Whispered Mumble into Low Mumble; Cough deliberately not folded in — a cough is a different sound), and it adds a third `no_burst` sub-type, `fp_span`: old-detector spans labelled Ahem / Low Mumble that no Gemini event overlaps — 1,115 from the real utterances and 1,200 mined the same way on DramaBox (locator v2 + old detector on 35,224 clips, so that fp_span is not learnable as "is real audio"). The negative mix per epoch is 0.55 speech / 0.20 silence / 0.25 fp_span (x2: 0.75 / 0.25). Everything else is x2's recipe verbatim: one grouped split (real by speaker, DramaBox by prompt), split seed 0, five initialisations, 120 epochs, 3584 → 256 → 19 (768 → 256 → 19), dropout 0.3, 25 held-out per class.

The two new classes are 96–97 % DramaBox material. Gemini's real-audio half has 44 Ahem and 14 Low Mumble segments (x2's rule of ≥ 100 per half would have admitted neither); they were admitted on the pooled count (1,099 / 418). Their real-audio recall below is measured on those few rows and flagged ⚠ accordingly. Re-annotating old Ahem / Low Mumble spans to get more real examples was costed and dropped: finding 1 puts the expected yield at ~1 %.

The number this repository exists for: rejection of the false-alarm spans

no_burst recall on held-out rows by sub-type (fp_span rows are speaker-/prompt-disjoint from training; x2 never trained on any fp_span row, so all 2,315 of them are fair for x2 — it rejects 0.2 %):

On the 1,967 stored spans of 3,200 validation clips (emolia + podcast, never seen by either head; */validation_spans_x2_vs_x3.md), gate P(no_burst) ≥ 0.5:

Sharp Inhale as top-1 goes from 70.2 % of spans (x2) to 1.2 % (large-v2) / 3.4 % (commercial). The rejection is flat across level-dip bands (≤ −20 dB: 94 %; > 0 dB: 91 %), i.e. it is not a level detector.

Is it rejecting the right spans? Verified against unverified

All 5,143 old spans of the 3,598 Gemini utterances, split by whether a Gemini event overlaps the span (*/rejection_verified_vs_unverified.md). "Unseen" = the utterance contributed no fp_span training row.

So the two heads trade differently: `production/` rejects more, including 14 % of verified bursts at gate 0.5; `commercial/` rejects almost only the `Ahem`/`Low Mumble`-type false alarms and keeps 97 % of verified bursts. P(no_burst) is delivered ungated, and the gate moves the trade-off (ablations/gate_sweep.json):

A reasonable default for production/ on corpus spans is a gate of 0.9; for commercial/, 0.5.

What the new negatives cost on burst classification

The old classes are the same 16 and go through the same admission rule; what changes their numbers is (a) two more classes to confuse with and (b) the fp_span negatives. Both were isolated (ablations/, large-v2, same split seed, x2 re-run reproduces x2's release exactly at 0.4659):

The balanced real-audio loss comes from the fp_span negatives, is consistent across initialisations, does not shrink when the mix share is cut, and lands on Frustrated Groan, Breathy Giggle, Exhausted Groan, Affirmative Grunt and Sharp Inhale — which move into their neighbour class (groan ↔ groan, giggle → chuckle), not into no_burst or the new classes. On the prior-weighted held-out sets (the rows that look like the corpus) nothing is lost. Across five split seeds (cross-source table below) the 17-class mean on real audio is unchanged (0.454 vs 0.449) and −2.7 pts on DramaBox, so the single-split −6 is partly that split.

If your use is grading generated bursts by class rather than cleaning corpus spans, the variant without fp_span negatives ships as ablations/large-v2_19cls_no_fp_span/ (same 19 classes).

Per class, x2 → x3

large-v2 (production/)

Strict recall on x3's held-out rows (n, x3, family), and the x2 → x3 change on the rows held out by both splits (n_c, x2_c → x3_c; neither head trained on them). ⚠ = fewer than 30 held-out rows in one source.

commercial (commercial/)

Strict recall on x3's held-out rows (n, x3, family), and the x2 → x3 change on the rows held out by both splits (n_c, x2_c → x3_c; neither head trained on them). ⚠ = fewer than 30 held-out rows in one source.

Cross-source 2×2 (five split seeds, balanced 25/class, one head per seed and arm)

Trained on the left, tested on the right; chance = 5.3 % over 19 classes (x2: 5.9 % over 17). The "17-class mean" column averages x3's per-class recall over the 16 old classes + no_burst so it is comparable to x2's number in the same row.

The real → arms have 19 real Ahem and no real Low Mumble training rows (all 14 are held out), so their Ahem/Low Mumble cells measure DramaBox-only or near-zero training. `Ahem` transfers from DramaBox to real speech at 0.25–0.31 (25 real rows per seed; chance 0.05). `Low Mumble` does not (≤ 0.10) and should be read as a DramaBox-only class until real examples exist.

How to call it

import sys; sys.path.insert(0, "<dir containing bs_score.py>")   # ships in production/
from bs_score import ProductionBurstScorer

sc  = ProductionBurstScorer(device="cuda", ckpt_dir="<.../production>")
out = sc.classify(cuts16)                    # spans you already cut: list[np.float32 @16 kHz]
# each: {"label", "prob", "labels": top-3, "family", "no_burst_prob"} -- apply your own gate
out = sc.bursts(wavs16, no_burst_gate=0.9)   # locate with laion/vocalburst-locator v2, then name

Same API and output shape as x2; the checkpoint carries classes (19, no_burst last), family, label_merge, extra_classes and neg_mix. commercial/ uses bs_vcsmall.py for the encoder and the same checkpoint format. Families: Ahem → throat, Low Mumble → hum.

What it cannot do

• —Its labels have never been checked by a human. Every training and test label is one model's opinion (gemini-3.8-flash, blind, whole utterances). The fp_span negatives are defined as "Gemini heard nothing there"; a quiet real breath that Gemini did not annotate is, by that standard, a non-event, and production/ will reject some real breaths (14 % of verified spans at gate 0.5, 6 % at 0.9).
• —`Low Mumble` is a DramaBox class. 14 real examples in total, real recall 0.00–0.14.
• —It is a classifier, not a locator. It labels a span something else proposed. On the 3,598 annotated utterances the old locator finds 42 of 44 Gemini Ahem events and 13 of 14 Low Mumble events — but only 23 % of Gemini's Sharp Inhale events and 47 % of Deep Breath.
• —Half its training data is TTS output (DramaBox, LTX-2 Community Licence not assessed), and the two new classes almost entirely so.

Files

• —production/ — large-v2 head: vocal_burst_mlp_prod_s{0..4}.pt, bs_score.py, bs_vclap.py, bs_det.py, prod_report.json, per_class_recall.{json,csv}, cross_source_2x2.json, validation_spans_x2_vs_x3.{md,json}, rejection_verified_vs_unverified.{md,json}.
• —commercial/ — the same for the 768-d head, bs_vcsmall.py.
• —ablations/ — large-v2_19cls_no_fp_span/ (heads + reports), large-v2_17cls_with_fp_span/, large-v2_17cls_no_fp_span_x2_rerun/, *_init5-9/, large-v2_19cls_fp_mix_0.10/ (reports), gate_sweep.json.
• —training_extras/ — fp_span_negatives.jsonl (2,315 rows: parent clip, span, old label, level dip, why), old_spans_vs_gemini_3598_utterances.json (finding 1), x2_on_gemini_ahem_lowmumble_segments.json (finding 3).
• —GROUPS.md, vocal_burst_groups.json — the 23-group scheme, unchanged from x2.