laion/vocalburst-locator

Vocal Burst Locator

A Whisper-based model that detects and localizes vocal bursts (laughs, coughs, sneezes, sighs, gasps, cries, screams, etc.) in audio, returning precise start/end timestamps for each event.

⭐ Start here: use model_v2.pt

The recommended default checkpoint is [`model_v2.pt`](https://huggingface.co/laion/vocalburst-locator/blob/main/model_v2.pt) (972 MB), fine-tuned on real in-the-wild audio. The original model.pt (v1) is trained on synthetic soundscapes only and is superseded — it is kept for reproducibility, documented under Previous version — v1.

⚠️ Two things are easy to get wrong, so they are stated up front:

1. 1.inference.py still auto-downloads `model.pt` when you do not pass a checkpoint. Pass model_v2.pt explicitly.
2. 2.inference.py's built-in post-processing defaults are still the v1-era values (threshold=0.65, merge_gap=0.3, min_dur=0.5). Pass the v2 values explicitly — they dominate the measured F1 (see below).

Recommended post-processing (v2)

threshold    = 0.50   # was 0.65 in v1
merge_gap    = 0.10   # was 0.30 in v1
min_duration = 0.10   # was 0.50 in v1   <-- the one that matters

Ground-truth bursts have a median duration of ~180 ms. A min_duration of 0.5 s therefore discards ~96 % of real bursts before matching. On one identical checkpoint, only changing post-processing moved event F1 from 0.243 to 0.598 — a larger effect than any training change made for v2. If you read older instructions in this card recommending 0.65 / 0.3 / 0.5, those are the v1 numbers and are not recommended any more.

Copy-pasteable usage

from huggingface_hub import hf_hub_download
from inference import load_model, detect_vocal_bursts   # inference.py from this repo

# 1. Download the recommended checkpoint
ckpt = hf_hub_download("laion/vocalburst-locator", "model_v2.pt")

# 2. Load it (v1 would be loaded if you omit `checkpoint`)
model, fe, device = load_model("cuda", checkpoint=ckpt)   # or "cpu"

# 3. Detect, with the v2 post-processing values
events = detect_vocal_bursts(
    "audio.mp3",
    model=model, fe=fe, device=device,
    threshold=0.50,
    merge_gap=0.10,
    min_dur=0.10,
)

for ev in events:
    print(f"{ev['start']:.2f}s - {ev['end']:.2f}s  (confidence: {ev['confidence']:.2f})")

Command line equivalent:

python inference.py audio.mp3 \
  --checkpoint "$(python -c 'from huggingface_hub import hf_hub_download; print(hf_hub_download("laion/vocalburst-locator","model_v2.pt"))')" \
  --threshold 0.50 --merge-gap 0.10 --min-dur 0.10 --device cuda

Raw state dict (if you build the model yourself — same WhisperSegmenter state dict as v1, 485 tensors, LoRA already merged):

import torch
sd = torch.load("model_v2.pt", map_location="cpu")
model.load_state_dict(sd)     # same keys as model.pt

Why v2 — measured on real audio

Re-measured on a held-out set of 992 real, in-the-wild expressive-speech clips, each checkpoint given a post-processing sweep to find its best possible operating point:

4.0x higher F1 on real audio. Note also how v1 fails: it only reaches usable precision at threshold 0.80, where recall collapses to 0.21 — on real recordings it is very unsure, and buying precision costs it four fifths of the events. v2 operates at 0.50 with recall 0.67.

Which checkpoint to pick

On real audio the two v2 weights are statistically indistinguishable; they differ only on the synthetic-soundscape domain. Both use the same post-processing values above. Details in the `model_v2_mixed.pt` section.

🔗 Ensemble: pair this detector with the captioner `laion/vocalburst-captioning-whisper` — locate bursts here, then caption each detected segment with that model. See the threshold study below.

Installation

pip install torch transformers soundfile librosa huggingface_hub

Expected output

Detected 3 vocal burst(s) in audio.mp3:

  1. 2.14s - 3.82s  (duration: 1.68s, confidence: 0.89)
  2. 8.50s - 9.12s  (duration: 0.62s, confidence: 0.74)
  3. 15.30s - 16.94s  (duration: 1.64s, confidence: 0.92)

JSON output (--json):

{
  "file": "audio.mp3",
  "events": [
    {"start": 2.14, "end": 3.82, "confidence": 0.89, "duration": 1.68},
    {"start": 8.5, "end": 9.12, "confidence": 0.74, "duration": 0.62},
    {"start": 15.3, "end": 16.94, "confidence": 0.92, "duration": 1.64}
  ]
}

Model Description

This model performs binary frame-level segmentation on audio: for each 20ms frame in a 30-second audio clip, it predicts whether a vocal burst is occurring. Post-processing then groups these frame-level predictions into discrete events with timestamps and confidence scores.

Architecture

Audio (16kHz, 30s) → Whisper-small Encoder (LoRA rank-8 merged) → 1500 frame embeddings
    → Linear(768→384) + GELU + Dropout
    → Conv1d(384, kernel=7) + GELU + Dropout   (temporal smoothing)
    → Linear(384→1) → sigmoid → 1500 probabilities
    → Post-processing → [(start, end, confidence), ...]

The model uses OpenAI's Whisper-small encoder as the audio feature backbone. During training, the encoder was adapted using LoRA (rank 8, alpha 16) on the q_proj and v_proj attention matrices. The LoRA weights have been merged into the base weights, so no adapter library is needed at inference time. All three checkpoints (model.pt, model_v2.pt, model_v2_mixed.pt) share this architecture and load with identical code.

Files

Inference Parameters

The "built-in default" column is what the script uses if you pass nothing; it has been left at the v1 values for backwards compatibility. Pass the recommended column explicitly.

Understanding Precision, Recall, and the Threshold Trade-off

Imagine the model is a security guard watching for vocal bursts. It has to make a decision for every moment of audio: "Is this a vocal burst, or not?"

There are four possible outcomes:

                        REALITY
                   Vocal Burst    Not a VB
                 ┌─────────────┬─────────────┐
  MODEL    Yes   │ True Pos ✓  │ False Pos ✗ │  ← "False alarm"
  SAYS:          │ (correct!)  │ (oops)      │
                 ├─────────────┼─────────────┤
           No    │ False Neg ✗ │ True Neg ✓  │  ← "Missed it"
                 │ (missed!)   │ (correct!)  │
                 └─────────────┴─────────────┘

• —Precision = Of everything the model flagged, how many were real? TP / (TP + FP)
• —High precision → when the model says "vocal burst!", it's almost always right
• —Low precision → lots of false alarms (the model is trigger-happy)

• —Recall = Of all real vocal bursts, how many did the model catch? TP / (TP + FN)
• —High recall → the model rarely misses a real event
• —Low recall → the model is too conservative, missing real events

• —F1 Score = The harmonic mean of precision and recall — balances both into one number.

How Each Parameter Affects Results

threshold — The confidence cutoff

The model outputs a confidence score (0 to 1) for every 20ms frame. The threshold decides: "How confident must the model be before we call it a vocal burst?"

low threshold   →  Model flags almost everything
                   ✓ High recall (catches most VBs)
                   ✗ Low precision (many false alarms)
                   Think: paranoid security guard

high threshold  →  Model only flags when very sure
                   ✓ High precision (almost no false alarms)
                   ✗ Low recall (misses quieter/ambiguous VBs)
                   Think: lazy security guard

For model_v2.pt the swept best operating point on real audio is 0.50. (For v1 on synthetic data it was 0.65; for v1 on real audio it was 0.80, where recall collapses — see Previous version.)

min_dur — Minimum event duration

After grouping confident frames into events, discard any event shorter than min_dur.

min_dur = 0.1s  →  Recommended for v2 on real audio
                   ✓ Keeps short coughs/gasps and the ~180 ms median real burst
                   ✗ Slightly more short false positives

min_dur = 0.5s  →  The old v1 default
                   ✓ Filters noise spikes in synthetic soundscapes
                   ✗ Discards ~96 % of real bursts

min_dur = 1.0s  →  Only keeps long events
                   ✗ Misses almost everything on real audio

This is the single most impactful knob. On synthetic soundscapes, mixed-in bursts are long (0.5–3 s) and a large min_dur cheaply removes false positives — which is why v1 shipped 0.5. On real recordings the ground-truth median burst is ~180 ms, so the same setting throws away the majority of true events.

merge_gap — Gap tolerance for merging

If two detected segments are separated by less than merge_gap, merge them into one event.

merge_gap = 0.0s  →  No merging. A laugh with a brief pause becomes 2 events.
                     Result: Over-counting (more events than expected)

merge_gap = 0.1s  →  Recommended for v2. Bridges frame-level dropouts without
                     swallowing neighbouring bursts.

merge_gap = 1.0s  →  Even 1-second gaps get bridged.
                     Result: Separate nearby events might merge into one big event

Because real bursts are short and can occur close together, a large merge_gap fuses distinct events; 0.10 s is the swept-best value for v2.

The Precision-Recall Trade-off (Why You Can't Have Both at 100%)

Making the model more cautious (↑ precision) always means it will miss more real events (↓ recall), and vice versa. You can't eliminate false positives without also losing some true positives.

             ← More conservative        More aggressive →

  Precision: ████████████████░░░░  (goes DOWN as you lower threshold)
  Recall:    ░░░░████████████████  (goes UP as you lower threshold)
                       ↑
               Sweet spot (F1 max)

Choose your trade-off based on your application:

• —Automatic subtitling: Prefer high precision (don't annotate noise as laughter)
• —Safety monitoring: Prefer high recall (don't miss a scream or cry for help)
• —Research/counting: Use balanced F1 (minimize both types of errors)

v2 — how it was trained

Same architecture, initialised from model.pt, then fine-tuned end-to-end (encoder unfrozen, encoder LR 1e-5, head LR 5e-4, linear schedule, BCE with pos_weight 2) on 98,296 real 30 s clips with CrisperWhisper-derived burst timestamps. An intermediate stage over ~1M additional windows was run and discarded — see below.

Post-processing matters more than the weights

The defaults published with v1 (threshold 0.65, merge_gap 0.3, min_duration 0.5) are badly mismatched to real data: ground-truth bursts have a median duration of 180 ms, so min_duration = 0.5 discards ~96 % of them before matching. On the identical checkpoint, sweeping post-processing moved event F1 from 0.243 to 0.598 — a larger effect than any training change we made. Recommended for v2:

threshold = 0.50      # was 0.65
merge_gap = 0.10      # was 0.30
min_duration = 0.10   # was 0.50  <-- the one that matters

A negative result worth recording

An intermediate fine-tuning stage over 1,044,713 windows cut from the same corpus hurt: F1 fell from 0.598 to 0.482. Cause: the window extractor kept only windows that contained at least one burst, so 100 % of that training set was positive. Without burst-free examples the detector learns that bursts are everywhere — precision fell from 0.649 to 0.578 and binary detection accuracy from 0.913 to 0.853. A subsequent stage on the balanced set recovered it to 0.607. If you train on your own data, keep negatives in.

model_v2_mixed.pt — broader domain coverage

A third weight, for the case where the audio is not only expressive speech. Same architecture and same loading code as the others.

model_v2.pt is fine-tuned on real expressive speech only and, in the process, forgot the synthetic-soundscape domain v1 was trained on — music beds, sound effects, non-speech backgrounds. model_v2_mixed.pt is trained on a mix: the regenerated v1 soundscape corpus (33,012 clips, 50 % burst-free by construction) plus 40,000 classifier-confirmed DramaBox clips.

Measured, each checkpoint at its own swept-best post-processing

Event F1 @ IoU 0.5.

Which to use. On real audio the two v2 weights are statistically indistinguishable — every difference sits inside the bootstrap confidence interval and the sign flips between validation sets. Do not read 0.607 vs 0.597 as a ranking. The one difference that is robust is the synthetic column: +0.21, CI [+0.16, +0.27].

• —annotating in-the-wild audio that includes music, SFX or non-speech → model_v2_mixed.pt
• —expressive speech only, and you want the weight that has been in use longest → model_v2.pt

Same post-processing recommendation for both: threshold 0.50, merge_gap 0.10, min_duration 0.10.

What did NOT work, so you don't repeat it

Three attempts to beat 0.607 on real audio failed. Training on 1,044,713 edge-case windows that were 100 % positive dropped F1 to 0.482; precision fell first, as a detector with no negatives learns that bursts are everywhere. A 100k positive / 100k negative "mirror" set — negatives made by excising the burst from the same clip — reached only 0.458, so simply restoring the positive/negative balance was not the fix either. The mix above is the first variant that does not lose ground, and it still does not gain any on real speech.

A hypothesis we tested and discarded: that the training labels were heavily contaminated, because a classifier pass rejected 50.41 % of the source burst detections. Controls showed that figure is mostly an artefact of the 300 ms cut length — feeding the same classifier 3,000 certainly real bursts truncated to 300 ms yields 51.3 % "no burst", against 13.7 % at full length. On the actual labels the rejection rate is 7.76 %. A paired control (identical clips and schedule, only the labels cleaned) moved F1 by −0.007 / +0.004 / +0.003 across three validation sets, every interval straddling zero. Label cleaning changed nothing measurable.

Honest limits

The real-audio validation sets are 992 and 500 clips, which cannot resolve differences below roughly ±0.03. Their labels come from an ASR model, not from human annotation, so the achievable ceiling is unknown — a model cannot score above the labels' own agreement rate. Whether 0.61 is near that ceiling or far below it has not been measured.

Previous version — v1 (model.pt)

Superseded by `model_v2.pt`. Kept for reproducibility and for the synthetic-soundscape domain; on real in-the-wild audio it scores event F1 0.152 versus 0.607 for v2.

v1 performance (synthetic evaluation)

Evaluated on 300 held-out synthetic soundscapes with the v1 inference settings (threshold=0.65, mergegap=0.3s, mindur=0.5s):

On that synthetic test set the model catches ~78% of vocal burst events with ~90% precision. That number does not transfer to real recordings — see the real-audio comparison.

v1 usage

from inference import load_model, detect_vocal_bursts

# omitting `checkpoint` auto-downloads model.pt (v1)
model, fe, device = load_model("cuda")
events = detect_vocal_bursts("audio.mp3", model=model, fe=fe, device=device)

python inference.py audio.mp3                       # v1 weights + v1 defaults
python inference.py audio.mp3 --checkpoint ./model.pt --threshold 0.7 --min-dur 0.3
python inference.py audio.mp3 --json

v1 threshold behaviour (validation set, synthetic)

v1 parameter recipes (synthetic-era guidance)

These recipes were tuned on synthetic soundscapes. For real audio with model_v2.pt, start from 0.50 / 0.10 / 0.10.

Using head_only.pt (v1 head)

If you already have Whisper-small loaded or want to use a different Whisper variant:

import torch
from transformers import WhisperModel

# Load your own whisper encoder
whisper = WhisperModel.from_pretrained("openai/whisper-small")
encoder_out = whisper.encoder(input_features=mel_features).last_hidden_state  # [B, 1500, 768]

# Load just the segmentation head
head_sd = torch.load("head_only.pt", map_location="cpu")
# head_sd contains: proj.0.weight, proj.0.bias, temporal.0.weight, temporal.0.bias, out.weight, out.bias
# Apply: proj → permute → temporal → permute → out → squeeze → sigmoid

v1 experiment results

We compared frozen encoder, LoRA rank 2/4/8 with the v1 post-processing (threshold=0.65, mergegap=0.3s, mindur=0.5s, pos_weight=2):

Key findings:

• —Raising detection threshold from 0.5→0.65 and tightening post-processing doubled F1 with zero retraining (on synthetic data)
• —LoRA rank-8 provided 3.15× improvement over the original baseline (F1: 0.239 → 0.752)
• —Precision improved from 24% to 90% — false positives dropped by ~90%
• —Diminishing returns above rank 8; rank 4 may be the sweet spot for cost/performance

Vocal-burst captioning ensemble & detection-threshold study (v1 post-processing)

This detector is designed to be used as an ensemble with the fine-tuned captioner `laion/vocalburst-captioning-whisper`: the locator finds where vocal bursts occur (start/end timestamps); each detected segment is then cut and described by the captioner (Whisper-small fine-tuned on vocal-burst captions). Together they turn raw audio into timestamped, captioned vocal-burst events that feed the LAION Universal Audio Annotation Pipeline.

⚠️ This study was run with merge_gap = 0.3 s, min_dur = 0.5 s — the v1 post-processing. Its threshold recommendation (0.85–0.89) is tied to those settings and does not carry over to model_v2.pt, where the recommended operating point is threshold 0.50, merge_gap 0.10, min_duration 0.10.

How the study was run

We swept the detector's confidence threshold from 0.85 to 0.92 (1% steps) on 150 audio samples (clean-speech false-positive checks + clips with inserted bursts + isolated bursts), with merge_gap = 0.3 s, min_dur = 0.5 s. For every (sample × threshold) the detector's segments were captioned by laion/vocalburst-captioning-whisper and the audio + (start, end, caption) list was sent to Gemini 3.1 Pro, which rated three axes 0–5 (5 = perfect): caption quality, timestamp accuracy, and completeness (do the detections cover ALL real vocal bursts, penalizing both misses and false positives). That is 1,200 independent LLM judgments; overall = mean of the three axes.

Results — average Gemini-3.1-Pro scores per threshold (ranked)

Findings: scores are tightly clustered across 0.85–0.92 (the detections change little in that band); threshold ≈ 0.88 is the sweet spot (best overall). Timestamp accuracy is consistently strong (~4.0), caption quality is moderate (~3.2), and completeness is the weakest axis (~3.0–3.15) — it degrades at the highest thresholds (0.91–0.92) as real bursts start being missed.

📊 Full interactive report (stats table + audio players + predictions + per-clip Gemini scores for the top-3 thresholds): `vocalburst_threshold_report.html`.

Training

Full Pipeline (v1 synthetic recipe)

# 1. Download source audio (~15K vocal bursts, ~13K backgrounds)
python download_sources.py

# 2. Generate synthetic soundscapes (~33K samples)
python generate_dataset.py

# 3. Train with LoRA (best v1 configuration)
CUDA_VISIBLE_DEVICES=0 \
  FREEZE_ENCODER=1 LORA_RANK=8 LORA_ALPHA=16 \
  POS_WEIGHT=2 DET_THRESHOLD=0.65 POST_MERGE_GAP=0.3 POST_MIN_DUR=0.5 \
  EPOCHS=15 LR=5e-4 ENCODER_LR=2e-4 \
  python train.py

For a v2-style run on real audio, initialise from a checkpoint with INIT_WEIGHTS, unfreeze the encoder, and set the eval/post-processing variables to the v2 values (DET_THRESHOLD=0.5 POST_MERGE_GAP=0.1 POST_MIN_DUR=0.1) — otherwise the reported eval metrics will be dominated by the mismatched POST_MIN_DUR.

Training Configuration

The training script is controlled entirely via environment variables:

Data Generation

The synthetic dataset generator creates audio soundscapes by mixing:

• —Vocal burst sources: ~15,680 clips from HuggingFace (laughs, coughs, sneezes, etc.)
• —Background sources: Music (5,000), AudioSet SFX (5,000), AudioSnippets (3,000)
• —Parameters: Random background type, 0-5 VBs per clip, varied SNR, up to 30s duration
• —Split: 50% positive (with VBs) / 50% negative (background only)

Each sample produces an .mp3 audio file and a .json metadata file:

{
    "events": [
        {"start_time": 3.21, "end_time": 4.85},
        {"start_time": 12.50, "end_time": 13.10}
    ],
    "duration_sec": 24.5,
    "bg_type": "music",
    "n_vocal_bursts": 2
}

Limitations

• —30-second maximum: The model processes 30s clips. For longer audio, segment into overlapping 30s windows.
• —Vocal burst types: Trained primarily on laughs, coughs, sneezes, sighs, gasps, cries. May not generalize to all vocal burst types.
• —Frame resolution: 20ms per frame (50 fps). Event boundaries are accurate to ±20ms.
• —Domain: model_v2.pt is fine-tuned on real expressive speech and has lost some of v1's synthetic-soundscape performance (0.513 vs 0.740 event F1 on synthetic); use model_v2_mixed.pt if music/SFX backgrounds matter.
• —Label provenance (v2): v2's real-audio training and validation labels come from an ASR model, not human annotation; the achievable ceiling is unknown.
• —Synthetic training data (v1): v1 was trained on synthetic mixtures only, which is why it scores event F1 0.152 on real in-the-wild clips.

Downstream note: classifier Slap Face false positives

When pairing this locator with laion/vocalburst-classifier-single in a detect-then-classify pipeline, note that the classifier over-predicts Slap Face as top-1 on in-the-wild speech. The recommended mitigation is to skip that label and take the runner-up class. See that model's README for details and a code snippet.

Citation

@misc{vocalburst-locator-2025,
    title={Vocal Burst Locator: Whisper-based Vocal Burst Segmentation},
    author={LAION},
    year={2025},
    publisher={HuggingFace},
    url={https://huggingface.co/laion/vocalburst-locator}
}

License

Apache 2.0