Modelpublic

laion/voiceclap-commercial-vocalburst-blend

sourceHugging Facecc-by-4.0updated 2d agoView on Hugging Face

1likes

Model Card

VoiceCLAP-commercial Vocal-Burst Blend Scorer

A tiny, self-contained model that predicts, from a short speech clip containing a vocal burst (laugh, sob, gasp, sigh, groan, scoff, etc.), a 0-10 score for how naturally that vocal burst blends into the surrounding speech.

It is designed as a fast, cheap proxy reward for evaluating expressive TTS / voice-acting systems: does the non-verbal burst sound like an organic part of the performance, or does it sound spliced-in, robotic, or emotionally mismatched?

The package is fully standalone. It bundles the frozen VoiceCLAP-commercial audio embedder (weights + custom modeling code) and the trained blend head, so no other repositories are fetched at inference time — everything loads from local paths.

The bundled encoder is the frozen VoiceCLAP-commercial backbone; its weights are included in this repository for offline inference.

What it predicts — the 0-10 rubric

Score	Meaning
0	The burst is disconnected: spliced/pasted-in, robotic, or the wrong emotion for the context (also assigned to clips with no genuine burst at all).
5	The burst fits the context but sounds performed / acted — mediocre delivery, audibly synthetic or stagey.
10	Totally organic: the burst is indistinguishable from a natural human reaction embedded in the speech.

Intermediate values interpolate between these anchors. The output is clamped to [0, 10].

Why the commercial embedding

This head sits on the 768-d VoiceCLAP-commercial audio embedding. That embedder is small (same 768-d width as VoiceCLAP-small) but much stronger for this task: the commercial-based head clearly beats the small-based one and comes within a hair of the ~4.7× wider large-v2 embedding, at a fraction of the cost.

Embedder	Head	Blend width	Val MAE ↓	Val corr ↑
VoiceCLAP-commercial (this repo)	`Linear(768,50)→GELU→Dropout(0.2)→Linear(50,1)`	768	2.057	0.625
VoiceCLAP-small	`Linear(768,50)→GELU→Dropout(0.2)→Linear(50,1)`	768	2.360	0.418
VoiceCLAP-large-v2	(large head)	3584	1.895	0.645

Takeaway: on the same 768-d budget, moving from the small to the commercial embedding drops MAE from 2.360 → 2.057 and lifts correlation from 0.418 → 0.625 — essentially matching large-v2 (1.895 / 0.645) at 1/4.7 the embedding width.

Bonus: one encode, two scores

The same VoiceCLAP-commercial audio embedding also feeds the genuineness predictor `laion/voiceclap-commercial-genuineness`. If you encode a clip once with the bundled commercial embedder you can run both heads on that single 768-d vector — blend naturalness (this repo) and burst genuineness — without a second forward pass through the encoder.

Architecture

16 kHz mono waveform
      │
      ▼
VoiceCLAP-commercial audio encoder   (frozen, 768-d embedding)
      │  encode_waveform → 768-d
      ▼
L2-normalize                           (unit-norm, as in training)
      │
      ▼
standardize:  z = (emb − μ) / σ        (μ, σ are 1×768 stats from training)
      │
      ▼
Blend head (MLP):
   Linear(768 → 50) → GELU → Dropout(0.2) → Linear(50 → 1)
      │
      ▼
blend score ∈ [0, 10]

—Embedder: VoiceCLAP-commercial, a dual-tower CLAP-style model. Only the audio tower (encode_waveform) is used and it is kept frozen. It is bundled in this repo under voiceclap_commercial/ and loaded locally via trust_remote_code=True — nothing is downloaded at inference time.
—Head: a small MLP (≈38k params) trained on top of the frozen, L2-normalized 768-d embeddings. The μ/σ standardization stats are stored inside the checkpoint.

Usage

python

from blend_model import CommercialBlendScorer

# Loads the bundled VoiceCLAP-commercial embedder + blend head, all local.
scorer = CommercialBlendScorer(pkg_dir=".", device="cpu")  # or device="cuda"

# Score a single wav (any sample rate / channel count; resampled to 16k mono):
score = scorer.score("clip_with_laugh.wav")
print(f"blend naturalness: {score:.2f} / 10")

# Score many at once:
scores = scorer.score_batch(["a.wav", "b.wav", "c.wav"])

# Score an in-memory waveform:
import torchaudio
wav, sr = torchaudio.load("clip.wav")
score = scorer.score_waveform(wav, sr)

Command-line:

bash

pip install -r requirements.txt
python example.py clip_with_laugh.wav
# -> blend score (0-10): 7.41

Files in this repo

File	Purpose
`blend_model.py`	`CommercialBlendScorer` + `BlendMLP` inference code.
`blend_head_commercial.pt`	Commercial blend head weights + μ/σ stats.
`voiceclap_commercial/`	Bundled frozen VoiceCLAP-commercial embedder (weights + modeling code).
`example.py`	Minimal CLI example.
`requirements.txt`	Python dependencies.

Limitations

—Frozen encoder caps accuracy. Quality is upper-bounded by what the frozen VoiceCLAP-commercial audio embedding captures; the head cannot recover information the embedding discards.
—Mid-range scores are hardest. The model is most reliable at the extremes (clearly organic vs. clearly spliced/absent). Scores around the middle of the scale (4-6) carry the most uncertainty.
—Use as a fast proxy, not a judge. This is a lightweight reward / filtering signal (e.g. for ranking or reward-shaping TTS outputs). It is not a replacement for a strong multimodal judge or human evaluation on high-stakes decisions.
—Domain. Trained on speech clips with vocal bursts; behaviour on music, noise-only audio, or non-speech is undefined.

License

CC-BY-4.0. You are free to share and adapt this model, including for commercial use, provided you give appropriate credit to LAION. See <https://creativecommons.org/licenses/by/4.0/>.

Created and released by LAION.