CoolFace
Modelpublic

laion/voiceclap-commercial-attribute-heads

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face
0likes17downloads
Model Card

VoiceClap Attribute-Regression Heads (61 dimensions)

61 tiny MLP regression heads on top of the frozen `laion/voiceclap-commercial` 768-d speech embedding. One head per attribute; each head reads a single 768-d embedding and emits that attribute's continuous score directly — no calibration step, no bucket decoding.

The heads cover 55 perceptual/emotional voice dimensions (Anger, Valence, Arousal, Warm_vs._Cold, …), 4 audio-quality scores, plus Age, Gender, duration and talking_speed.

Headline results (held-out validation, 10 % stratified split)

bandcount
strong (Pearson r > 0.6)40
medium (0.3 ≤ r ≤ 0.6)21
weak (r < 0.3)0
median r across 61 dims0.683

Best: duration 0.998 (see caveat 1 — this one is trivial), score_overall_quality 0.949, Gender 0.940, score_background_quality 0.935, score_content_enjoyment 0.933, Monotone_vs._Expressive 0.901, Serious_vs._Humorous 0.896, High-Pitched_vs._Low-Pitched 0.894, Recording_Quality 0.893, Amusement 0.890, Arousal 0.822, Age 0.792, talking_speed 0.726, Valence 0.645. Weakest: Awe 0.338, Relief 0.355, Fear 0.390.

The full 61-row table is further down, and also as `report/results.csv` / `report/results.json`. An interactive report with per-dimension scatter plots is in `report/attrdistill_report.html`.

External benchmarks are reported [below](#benchmarks) — these are validation numbers on the training distribution, and they do not transfer unchanged.


Read this before you use the numbers

These four caveats are not footnotes; they change how the table should be read.

1. duration r = 0.998 is trivial and is not evidence of embedding quality

Waveform length is directly visible to the encoder, so predicting it is close to reading it off. It also saturates at 30 s, because that is the encoder window used everywhere in this project (the training target was clipped to 30 s as well). It is included as a pipeline sanity check, nothing more. Ignore it when judging how good the representation is.

2. Bucket-balanced training pushes the predicted mean up on rare emotions

Training sampled inversely to bucket frequency so that rare high-intensity examples were not drowned out. On zero-inflated dimensions this systematically shifts the predicted level upward. The worst offenders (see the bias = mean(pred) − mean(true) column): Thankfulness_Gratitude +0.65, Relief +0.49, Distress +0.44, Impatience_and_Irritability +0.42, Awe +0.41, Astonishment_Surprise +0.40, Infatuation +0.40, Disappointment +0.39, Anger +0.39, Triumph +0.38, Affection +0.38, Sadness +0.37, Fear +0.35.

Ranking within a dimension is good; the absolute level runs high. If you need a calibrated absolute value, subtract the bias column, or (better) fit a one-off affine map on a small sample of your domain. Do not threshold a raw score at "0.5 means present" without recalibrating.

3. On sparse emotions, Spearman ρ is the stricter and more honest read

Pearson r on a zero-inflated target is inflated by the easy separation of "lots of zeros" from "a few large values". Where r and ρ disagree, trust ρ: Anger r = 0.728 but ρ = 0.516; Malevolence_Malice 0.679 → 0.466; Distress 0.481 → 0.289; Pain 0.392 → 0.199; Sadness 0.503 → 0.345.

4. talking_speed is characters per second, not words per second

It reproduces the source dataset's talking_speed_5.00_to_25.00 bucket definition. Typical English conversational speech lands around 12–17 chars/s. Divide by ~5.5 for a rough words-per-second figure.


Reproducibility note that matters more than any hyperparameter: de-duplicate before you split

The training dataset files each clip under one WebDataset tar per dimension, so a single clip appears in up to 15 different tars. The raw tar-entry count is 504,007; the number of unique clips is 296,422.

If you build the train/val split on raw tar entries, the same clip lands in both train and validation, and every metric in every table becomes inflated — the model is being asked to recall an embedding it has already memorised. De-duplication is done here by WebDataset key, before the split, and it is the single most important step to reproduce these numbers rather than better-looking fake ones.

(Related quirk handled the same way: the Jealousy___Envy_* and Jealousy_and_Envy_* tars both carry the single score key Jealousy_&_Envy, so those two near-duplicate tar dimensions merge into one head. That is why 328 tars over ~15 dimension families yield 61 — not 62 — heads.)


Usage

Both files below are in this repo: `voiceclap_heads.py` is a ~180-line reference implementation, `example_inference.py` is the demo whose real output is pasted underneath.

bash
pip install torch torchaudio transformers huggingface_hub numpy

Single clip

python
from voiceclap_heads import AttributeScorer

scorer = AttributeScorer()          # laion/voiceclap-commercial + heads.pt
s = scorer.score("clip.wav")        # dict: 61 dimension name -> float

print(s["Valence"], s["Arousal"], s["Gender"], s["Recording_Quality"])

Batched (one encoder pass for many clips)

python
paths = ["a.wav", "b.wav", "c.wav"]

# all 61 heads, or a subset:
out = scorer.score(paths, dims=["Valence", "Arousal", "Anger"], batch_size=32)
# {'Valence': array([...]), 'Arousal': array([...]), 'Anger': array([...])}

# or keep the embeddings and re-run heads later (they are the expensive part):
import numpy as np
from voiceclap_heads import load_audio
emb = scorer.embed([load_audio(p) for p in paths], batch_size=32)   # (N, 768)
scores = scorer.score_embeddings(emb)                               # 61 x (N,)

If you already have VoiceClap embeddings

python
import torch
from voiceclap_heads import build_head

b = torch.load("heads.pt", map_location="cpu", weights_only=False)
head = build_head(b["heads"]["Arousal"])
x = (emb - b["emb_mu"]) / b["emb_sd"]      # emb: (N, 768) raw encoder output
arousal = head(x).squeeze(-1)              # the score, directly

Inference contract (must match, or the numbers do not transfer)

  1. 1.mono, resampled to 16 kHz
  2. 2.truncated to 30 s (480,000 samples) — the encoder window
  3. 3.batch padded to that batch's max length, not a fixed 30 s (voiceclap_heads.AttributeScorer.embed sorts by length and does this for you; padding policy has a small but real effect on the embedding, so keep it identical)
  4. 4.VoiceClap.encode_waveform(...) → (N, 768)
  5. 5.x = (emb - emb_mu) / emb_sd — both vectors ship inside heads.pt
  6. 6.head(x) → the score. The y-normalisation affine is already folded into each head's last `Linear`, so there is nothing to de-standardise afterwards.

Score scales and polarity

Scores are on the source annotation scale, which is roughly 0–7 for emotions (0 = not present) — but each dimension has its own empirical range. report/results.csv carries label_min, label_max, label_mean, label_p50, label_p99 and frac_below_0p5 per dimension; config.json carries y_min/y_max/y_mean/y_std per head. Heads are unbounded regressors — clip to the label range if you need it.

Polarity of the non-obvious dimensions, determined empirically (not assumed) by correlating our heads against the independent `laion/voicenet-dimension-predictors-commercial` predictors on 61 held-out clips:

dimensiondirectionevidence
Genderhigher = more femininer = −0.90 vs VoiceNet GEND (whose level 0 = "hyper-feminine", level 6 = "hyper-masculine")
Agehigher = olderr = +0.52 vs VoiceNet AGEV (level 0 = infant → 6 = elderly)
X_vs._Y nameshigher = more Ye.g. High-Pitched_vs._Low-Pitched high ⇒ lower-pitched; Monotone_vs._Expressive high ⇒ more expressive
durationseconds, capped at 30
talking_speedcharacters/second
Background_Noisehigher = more noiser = −0.29 with score_background_quality, −0.46 with Recording_Quality; ρ = +0.60 against emolia's BKGN rubric (whose level 6 = "noise overpowers the voice")
Vulnerable_vs._Emotionally_Detachedhigher = more detachedρ = +0.30 against VoiceNet VULN after sign flip (level 0 = "impenetrable, heavily armored", level 6 = "profoundly raw")

Worked example — real output

Run on this machine with python example_inference.py examples/clip_a.mp3 examples/clip_b.mp3 (CPU, ~2.7 s for both clips after model load). The two clips ship in `examples/`. They are LAION-generated studio-style speech and were not in the training set.

  • —clip_a.mp3 — generated from the direction "A 35-year-old female with a warm, resonant, slightly husky timbre … relaxed, a subtle smile playing on her lips", 22.15 s
  • —clip_b.mp3 — generated from the direction "A 78-year-old man with a gravelly, slightly strained voice … weary but resolute", 24.07 s
text
loaded 61 heads on cpu

==============================================================================
examples/clip_a.mp3
==============================================================================
-- speaker / voice
   Gender                                  -0.390
   Age                                      2.587
   High-Pitched_vs._Low-Pitched             1.749
   Monotone_vs._Expressive                  2.584
   Soft_vs._Harsh                           0.213
   Warm_vs._Cold                            0.685
   Confident_vs._Hesitant                   1.470
   Submissive_vs._Dominant                  0.758
   Serious_vs._Humorous                     1.695
   Vulnerable_vs._Emotionally_Detached      1.016
   talking_speed                            9.016
   duration                                22.266
-- recording quality
   Recording_Quality                        2.650
   Background_Noise                         0.958
   Authenticity                             3.255
   score_overall_quality                    2.941
   score_speech_quality                     1.861
   score_background_quality                 3.653
   score_content_enjoyment                  5.271
-- core affect
   Valence                                  0.574
   Arousal                                  3.210
-- top 8 emotion dimensions
   Elation                                  2.375
   Interest                                 2.352
   Hope_Enthusiasm_Optimism                 2.210
   Pleasure_Ecstasy                         1.831
   Amusement                                1.719
   Triumph                                  1.544
   Astonishment_Surprise                    1.540
   Distress                                 1.450

==============================================================================
examples/clip_b.mp3
==============================================================================
-- speaker / voice
   Gender                                  -0.910
   Age                                      3.199
   High-Pitched_vs._Low-Pitched             2.130
   Monotone_vs._Expressive                  1.945
   Soft_vs._Harsh                           0.301
   Warm_vs._Cold                            0.314
   Confident_vs._Hesitant                   0.934
   Submissive_vs._Dominant                  0.465
   Serious_vs._Humorous                     1.115
   Vulnerable_vs._Emotionally_Detached      1.863
   talking_speed                            8.629
   duration                                24.243
-- recording quality
   Recording_Quality                        2.528
   Background_Noise                         0.652
   Authenticity                             2.830
   score_overall_quality                    3.045
   score_speech_quality                     1.892
   score_background_quality                 3.823
   score_content_enjoyment                  5.025
-- core affect
   Valence                                  0.146
   Arousal                                  1.645
-- top 8 emotion dimensions
   Interest                                 2.210
   Concentration                            1.747
   Contemplation                            1.131
   Confusion                                1.089
   Astonishment_Surprise                    1.034
   Amusement                                1.005
   Impatience_and_Irritability              0.839
   Disappointment                           0.809

batched: {'Valence': [0.574, 0.146], 'Arousal': [3.21, 1.645], 'Gender': [-0.39, -0.91], 'Age': [2.587, 3.199]}

Reading it honestly: duration is recovered to within 0.12 s / 0.17 s; Age and Gender order the two speakers the right way (older & more masculine for clip_b); the older, weary clip gets much lower Arousal (1.65 vs 3.21) and lower Valence. Age 2.587 vs 3.199 is a 7-point scale, not years — it is not claiming 35 vs 78.

A caution on that example. The prompt text is a generation direction, not a measurement of the produced audio. On a 61-clip sample of the same pool, our Gender head correlates r = 0.09 with the gender stated in the prompt and Age correlates r = 0.05 with the stated age — while agreeing strongly with VoiceNet's independent predictors on the same audio (|r| = 0.90 and 0.52). The synthesis simply does not reliably follow the prompt's speaker description on this pool, so prompts are not usable as ground truth. Use the examples to check that your pipeline is wired correctly, not as an accuracy claim.

Training

  • —Data: `TTS-AGI/Emotion-Voice-Attribute-Reference-Snippets-DACVAE-Wave` — 328 WebDataset tars, 504,007 raw entries → 296,422 unique clips after key de-duplication (18,592 from the podcast-conditioned source, 277,830 from the emotion-attribute source).
  • —Targets: every sample's JSON carries the full 59-score empathic_insight_scores / annotation_scores vector, so every clip labels every dimension and the regression targets are exact continuous scores — not bucket midpoints reconstructed from the tar name. That is why n_train is ~266.8 k for essentially every head. Plus duration and talking_speed derived from the JSON.
  • —Embeddings: computed once with the frozen encoder into a single (296422, 768) float16 array; the heads then train in minutes.
  • —Loss: Huber (δ tuned per dimension over {0.5, 1.0}).
  • —Split: 10 % validation, stratified per score bucket, taken after de-duplication.
  • —Sampling: bucket-balanced (weight ∝ 1/bucket count) — see caveat 2.
  • —Feature standardisation: per-dimension mean/std over a 200 k random subsample, shared by all heads (emb_mu, emb_sd in heads.pt).
  • —Sweep: 19 runs per dimension — {mlp1_h64, mlp1_h128, mlp2_h96_64, mlp2_h64_32} × lr {1e-3, 3e-3} × wd {1e-4, 1e-2} (16 runs), then 3 dropout / Huber-δ refinement runs on the winning architecture. Selected by validation Pearson r, with cosine schedule, 200-step warmup, early stopping (patience 8 evals).
  • —Training script as run: `train_heads.py`.

Architectures

kindtopology
mlp1 (mlp1_h64, mlp1_h128)Linear(768,H) → GELU → Dropout → Linear(H,1) — same topology as VoiceNet's MLPHead
mlp2ln (mlp2_h96_64, mlp2_h64_32)Linear(768,H) → LayerNorm → GELU → Dropout → Linear(H,H2) → GELU → Dropout → Linear(H2,1)

Selected per dimension: mlp2_h96_64 ×30, mlp2_h64_32 ×23, mlp1_h128 ×7, mlp1_h64 ×1.

Files

pathwhat
heads.ptthe bundle: {"heads": {dim: state_dict}, "emb_mu": (768,), "emb_sd": (768,)} — 61 heads, 17 MB
heads_per_dim/<Dimension>.ptthe same heads individually, each with {"state_dict", "kind", "arch", "meta"}
config.jsonarchitecture kinds, per-dimension hyperparameters + metrics + label ranges, preprocessing contract, I/O spec
voiceclap_heads.pyreference implementation (AttributeScorer, build_head, load_audio)
example_inference.pythe demo above
examples/clip_a.mp3, clip_b.mp3the two clips used in the demo
report/results.md, results.csv, results.jsonthe full 61-row result table
report/attrdistill_report.htmlinteractive report, per-dimension scatter plots
train_heads.pythe training script, as run
bench/emolia/emolia-bench evaluation: endpoint server, run script, analysis scripts, the harness's own report + summary + per-item predictions
provenance/MANIFEST.jsonembedding-matrix manifest: n samples, label column order, source counts, de-dup notes
bench/emonet/EmoNet Voice Bench evaluation: script, per-row predictions, full metrics JSON, label table, data manifest
provenance/meta_all.parquetone row per unique clip: key, tar, dim, bucket, src, dur, nchar, nword — enough to reproduce the de-duplication and the split

The 296,422 × 768 embedding matrix and the 296,422 × 61 label matrix are not uploaded (0.5 GB). Regenerate them from the dataset with the contract above, or ask.

Intended use and limits

Fast, cheap attribute tagging, dataset filtering/curation, TTS-output QA, retrieval and reward shaping over large speech corpora — anywhere you want 61 numbers per clip at roughly the price of one embedding.

Limits.

  • —Training labels are model-generated (Empathic Insight annotations), not human ratings. These heads distil an annotator model; they inherit its biases, and agreement with human raters is an open question — see the benchmark section.
  • —Training audio is largely synthetic / TTS-generated English studio-style speech. Expect degradation on real-world noisy recordings, on non-English speech, and on spontaneous conversational audio.
  • —Medium-band emotions (r 0.3–0.6) are usable for ranking and coarse filtering, not for per-clip assertions about a speaker's emotional state.
  • —Do not use these to make consequential judgements about individuals (hiring, policing, clinical, insurance). Gender and Age are perceived-voice attributes, not identity claims, and are wrong often enough to be harmful if treated otherwise.

Citation

bibtex
@misc{voiceclap_attribute_heads_2026,
  title  = {VoiceClap Attribute-Regression Heads: 61 linear-probe dimensions over a frozen commercial speech encoder},
  author = {LAION},
  year   = {2026},
  url    = {https://huggingface.co/laion/voiceclap-commercial-attribute-heads}
}

Base encoder: laion/voiceclap-commercial. Training data: TTS-AGI/Emotion-Voice-Attribute-Reference-Snippets-DACVAE-Wave.


Benchmarks

Two external benchmarks were run against these heads. Neither is a regression benchmark shaped exactly like our training task, so read the methodology before the numbers.

1. EmoNet Voice Bench — human-expert emotion-intensity regression

Dataset: `t1a5anu-anon/emonet-voice-bench` (public, ungated) — 12,600 items, 12,397 unique clips, 35.8 h. Paper: EmoNet-Voice, arXiv:2506.09827.

Task, verbatim from the dataset. Each row carries exactly one target emotion and an intensity rated 0 = Not Present / 1 = Mildly Present / 2 = Intensely Present by 2–4 of 6 psychology experts. It is not a 40-way argmax and not multi-label. The paper maps the averaged human intensity to a 0–10 scale (0 → 0, 1 → 5, 2 → 10) and reports MAE, RMSE, Pearson r and Spearman ρ.

Our protocol. For each row we take the single head named by that row's emotion and score the clip with it — no argmax, no head selection. 42 of 42 benchmark classes map 1:1 onto a head, with only cosmetic renames (Astonishment→Astonishment_Surprise, Fatigue→Fatigue_Exhaustion, Hope→Hope_Enthusiasm_Optimism, Intoxication→Intoxication_Altered_States_of_Consciousness, Malevolence→Malevolence_Malice, Pleasure→Pleasure_Ecstasy, Thankfulness→Thankfulness_Gratitude, Emotional Numbness, Jealousy & Envy, Sexual Lust, Impatience and Irritability). The remaining 19 heads are unused. All 12,600 rows were scored; refusal rate 0 %.

Results

aggregationPearson rSpearman ρ
pooled over all 12,600 rows0.3040.323
macro-averaged over the 42 emotions0.3860.374
macro, restricted to the 2,912 rows with unanimous annotators0.5480.488

Published baselines (paper Table 5), for orientation:

modelSpearman ρPearson rMAERMSErefusal
EmpathicInsight-Voice Large0.4150.4212.9953.7560 %
EmpathicInsight-Voice Small0.4180.4142.9973.7570 %
Gemini 2.5 Pro0.4170.4163.0083.7850 %
Gemini 2.0 Flash0.3550.3503.6084.4530.01 %
GPT-4o Audio Preview0.3370.3363.4324.24727.6 %
GPT-4o Mini Audio Preview0.3260.3273.3204.1242.3 %
Hume Voice0.2740.2314.7445.47439.2 %
these heads (pooled)0.3230.304see belowsee below0 %

Honest reading: on the fair, scale-free comparison these 61-head probes land below Gemini 2.0 Flash and the EmpathicInsight-Voice models, above Hume Voice, and roughly level with GPT-4o Mini Audio Preview — while being a 50 k-parameter MLP on a frozen embedding rather than a large audio LLM. The paper does not state whether its Table 5 aggregates pooled or macro-over-emotions; we give both, and we quote our pooled figure in the table above because it is the less flattering of the two.

MAE / RMSE need calibration and are therefore not comparable

Our heads emit the Empathic-Insight 0–7 scale, not 0–10, and the benchmark's rows are heavily emotion-present (gold mean 5.26 on the 0–10 scale) while our raw predictions average 1.13. Raw MAE is dominated by that scale mismatch, not by ranking quality:

variantMAERMSEfair to compare?
fixed rescale clip(score/7×10, 0, 10), no fitting3.8844.587yes, but it is mostly measuring the scale gap
one global affine, 2-fold cross-fitted on the benchmark2.2892.786no — uses benchmark labels
per-emotion affine, 2-fold cross-fitted on the benchmark2.1422.607no — uses benchmark labels
per-emotion affine fit on all data (oracle)2.1242.579no — upper bound only

The calibrated rows would "beat" every model in Table 5. They are not a win. The LLM baselines are zero-shot and receive no in-domain calibration; we do. We report them only to show that most of the raw MAE gap is an offset/scale artefact rather than a ranking failure. Use the correlation table for comparisons. This is caveat 2 (the bucket-balanced bias) showing up in a second domain, in the opposite direction: our absolute level is far too low here because the benchmark over-samples present-emotion clips relative to our training marginal.

Presence detection (a secondary, non-official metric)

Reframing as "is this emotion present at all?" — clips all annotators rated 0 vs clips all annotators rated > 0 — gives a macro ROC-AUC of 0.810 over the 38 emotions with ≥ 10 clips on both sides. This is not the benchmark's defined metric; it is reported because it is the shape most downstream users actually want (filtering a corpus), and because it is threshold-free.

Contamination check

Identifier spaces are disjoint: benchmark clips are named <8-hex>_enhanced*.mp3, our training keys are cond_podcastt_* / emo_<dim>_b<bucket>_batch*_chunk*. Zero overlap on exact stems and zero overlap on 8-hex tokens. That is evidence of separate provenance, not proof of zero audio overlap — no audio-fingerprint dedup was run.

A lineage caveat that is more important than the filename check: our training labels are Empathic Insight model annotations, and the EmpathicInsight-Voice models were themselves trained on EmoNet-Voice data. So our teacher may have seen clips from the same parent corpus as this benchmark. Treat these numbers as distillation transfer measured against human experts, not as a clean held-out generalisation test against an unrelated corpus. The gap between our 0.30 and EmpathicInsight-Voice's 0.42 is roughly the cost of distilling that annotator into a 50 k-parameter probe.

Per-emotion results

presence AUROC is blank where one side had < 10 clips.

emotionnPearson rSpearman ρr (unanimous)presence AUROC
Teasing3000.6520.6430.7620.938
Impatience_and_Irritability3000.6510.6400.8060.940
Malevolence_Malice3000.5820.5590.7870.922
Authenticity3000.5770.4640.8190.939
Anger3000.5670.5720.7200.876
Amusement3000.5350.5080.7990.903
Pain3000.4990.4780.6760.826
Distress3000.4950.4970.6130.829
Sexual_Lust3000.4850.4570.5960.847
Contempt3000.4610.4530.6660.895
Disgust3000.4570.4600.7010.865
Helplessness3000.4540.4380.6310.856
Sadness3000.4400.3830.6500.820
Elation3000.4380.4460.6050.820
Fear3000.4130.4020.6750.836
Jealousy_&_Envy3000.4060.3810.5560.791
Fatigue_Exhaustion3000.3970.4280.5860.836
Pride3000.3960.3680.6250.811
Embarrassment3000.3800.3790.4070.752
Arousal3000.3700.3390.5520.823
Astonishment_Surprise3000.3680.3570.4880.744
Confusion3000.3670.3640.6230.869
Bitterness3000.3600.3300.514—
Infatuation3000.3580.3770.3670.782
Pleasure_Ecstasy3000.3550.3440.5520.745
Longing3000.3510.3630.5440.820
Affection3000.3420.2870.5530.789
Intoxication_Altered_States_of_Consciousness3000.3420.3570.4460.795
Sourness3000.3400.3500.539—
Shame3000.3350.3020.5070.792
Doubt3000.3150.3240.3820.735
Disappointment3000.3090.2890.4650.763
Thankfulness_Gratitude3000.3090.2500.5050.721
Interest3000.2890.2610.4850.843
Contentment3000.2840.3030.4520.786
Concentration3000.2770.2760.334—
Relief3000.2770.2630.4520.735
Triumph3000.2740.2900.4410.760
Contemplation3000.2620.2380.547—
Awe3000.1770.1530.2810.663
Emotional_Numbness3000.1710.2170.1390.754
Hope_Enthusiasm_Optimism3000.1080.1340.1510.569

Run with `bench/emonet/eval_emonet.py`; raw per-row predictions and the full metrics JSON are in `bench/emonet/`.

2. EmoIA / emolia-bench — binary "is this attribute present?" judgement

Benchmark: `LAION-AI/emolia-bench` (commit 1a48a7a). Two subsets, both binary classification of (audio, text-prompt) pairs against majority_present from 2–4 human raters:

  • —`emolia-emo` — 7,988 items over 3,944 clips, 40 emotions, 5 sampling strata.
  • —`emolia-dim` — 18,632 items over the 58-dimension VoiceNet rubric (58 dims × 7 levels × {positive, negative}).

Official metric: balanced accuracy vs majority_present (benchmark.py::binary_metrics, bal = (sensitivity + specificity)/2); secondary metric positive-class F1. We ran the benchmark's own harness through its --endpoint path — a local server returns the relevant head's score as the "similarity" — with the README's recommended filter --min-raters 2 --exclude-flagged.

2a. emolia-emo — ran in full, 40/40 emotions map to a head

n = 7,986 after filtering (2 items dropped for having < 2 raters; --exclude-flagged dropped 0, because this subset's benchmark_labels.csv has no flagged column at all — that filter is a silent no-op here, not evidence of clean audio). Positive rate 0.578. 0 missing audio, 0 evaluation errors. Only one name needed mangling: the benchmark's Jealousy / Envy → head Jealousy_&_Envy. 21 of the 61 heads are unused.

Threshold. The harness defaults to --threshold 0.0, which assumes a cosine-style score centred on zero. Our heads emit a non-negative 0–7 intensity, so that default is degenerate (balanced accuracy 0.515 — it calls almost everything present). We used `--threshold 1.0`, chosen a priori as the Empathic-Insight "mildly present" boundary, with no reference to benchmark labels. For transparency: the oracle-tuned threshold is 0.919 and yields 0.696, so the a-priori choice cost ~0.002.

balanced accuracyF1
all 7,986 items, threshold 1.0 (headline)0.6940.696
unanimous-rater subset (n = 2,614)0.8220.867
harness default threshold 0.00.5150.734
oracle-tuned threshold 0.919 (not a headline)0.6960.713

Baselines the benchmark itself computes on the identical task:

referencebalanced accuracy
random / always-majority0.500
single human, pairwise0.562
single human, leave-one-out vs the majority of the others (n = 13,208)0.572
these heads0.694

By the benchmark's own rubric that is the "Medium — above single-human, useful training target" band.

The 0.694 is inflated, and here is the measurement that shows it

emolia-emo's item pool was selected using an earlier version of these very heads: every sidecar JSON under dataset/emolia-emo/data/<Emotion>_best/ carries 55 <Dimension>_best columns from this head family, and the folder name means "top-scoring clips for that head". The human labels are independent, so this is selection bias, not label leakage — but it is home turf, and the pooled number is not a clean estimate.

The benchmark's 5 task_type strata let us measure how much. Within a stratum the selection confound is held roughly constant:

task_typenbalanced accAUROC
affirmative3,9990.6290.706
contrastive_11,0000.5350.559
contrastive_21,0000.5390.556
penultimate9920.5260.562
ultimate9950.5010.576
pooled7,9860.6940.748
mean within stratum0.5460.592

Roughly half the pooled signal is the selection confound. The honest headline is the mean within-stratum AUROC of 0.592 — modest, above chance, and far below what the pooled 0.748 suggests. On the hard contrastive/ultimate strata — where the negatives are near-misses rather than random other emotions — these heads are close to chance.

A useful control: scoring the same items with the sidecar's own older head values gives pooled AUROC 0.749 and mean within-stratum AUROC 0.573, versus 0.748 / 0.592 for the heads published here. The released heads are marginally better and behave essentially like the selector, which is exactly what the selection-bias story predicts.

Note also that the benchmark's own inter-rater agreement is low (Fleiss κ ≈ 0.086 binary on emolia-emo; single-human LOO balanced accuracy 0.572), so there is little headroom above ~0.6 for anyone on the within-stratum reading.

2b. emolia-dim — not run as a headline number, and here is why

We deliberately do not report a balanced accuracy for emolia-dim. Three independent reasons, any one of which would be disqualifying:

  1. 1.Half the benchmark has no corresponding head. 30 of 58 dimensions — 9,360 rows, 50.2 % — have no head at all: ARSH, ATCK, CHNK, DARC, DFLU, EMPH, EXPL, FULL, LANG, METL, RESP, R_CHST, R_HEAD, R_MASK, R_MIXD, R_NASL, R_ORAL, R_THRT, SMTH, STRU, S_CONV, S_MONO, S_NARR, S_NEWS, S_STRY, S_TECH, VALS, VFLX, VOLT, WARM.
  2. 2.Some are structurally unreachable, not merely missing. The 7 resonance dimensions (R_CHST/THRT/ORAL/MASK/NASL/HEAD/MIXD) and the 6 within-clip dynamics dimensions (VOLT, VFLX, DARC, ARSH, VALS, ATCK) describe trajectory inside the clip. Our heads emit one clip-level scalar. No calibration fixes that; it needs a windowed head.
  3. 3.The task shape does not fit a monotone scalar. emolia-dim asks "does this clip sit at level L?", not "how much of D?". A monotone score thresholded once globally cannot answer bucket membership. (This is the same reason CLAP baselines struggle on this subset — the harness was built for cosine similarity.)

WARM deserves a specific flag: it is a false friend. The rubric defines it as spectral warmth (200–500 Hz low-mid energy); our Warm_vs._Cold is affective / interpersonal warmth. Same word, different construct — so we list it as "no head" rather than claim the match.

What we do report — a diagnostic, not the benchmark's metric. For the 8 dimensions that genuinely share a construct with a head, Spearman ρ between the head score and the queried rubric level, restricted to items humans agreed sit at that level. This measures monotone agreement with the rubric's ordering:

VoiceNet dimheadsignnSpearman ρ
GEND Perceived GenderGender−660.681
BKGN Background NoiseBackground_Noise+790.597
VALN ValenceValence+890.589
STNC StanceSubmissive_vs._Dominant+970.518
AROU ArousalArousal+750.450
TEMP Tempotalking_speed+640.441
RCQL Recording QualityRecording_Quality+780.430
VULN VulnerabilityVulnerable_vs._Emotionally_Detached−890.302
macro mean0.501

Sign conventions were read off the rubric text, not fitted: GEND level 6 = hyper-masculine while our Gender rises toward feminine; VULN level 0 = "impenetrable, heavily armored" and level 6 = "profoundly raw, naked", while our head rises toward Emotionally Detached. One trap worth recording: emolia's `BKGN` rubric runs opposite to the one in `voicenet .../dimensions.py` — here level 0 is a "dead acoustic vacuum" and level 6 is "noise overpowers the voice", so +1 is correct for this rubric and would be wrong for the other. Sample sizes are small (64–97 items per dimension); treat these as indicative.

As an additional sanity figure only: applying a label-free quantile-matching calibration (map each head score onto 0–6 by matching its empirical quantiles to the level marginal, then predict "yes" iff the mapped level equals the queried level) gives a macro balanced accuracy of 0.548 over those 8 dimensions. It covers 8 of 58 dimensions and is not comparable to a full-benchmark number.

Artifacts, scripts and the harness's own report.md / summary.json are in `bench/emolia/`.

Benchmarks — what to take away

  • —On human-rated emotion-intensity regression (EmoNet Voice Bench) these probes reach pooled r = 0.304 / ρ = 0.323 with a 0 % refusal rate: below Gemini 2.0 Flash and the EmpathicInsight-Voice models, above Hume Voice, at ~50 k parameters per dimension.
  • —On binary attribute presence (emolia-emo) the pooled 0.694 balanced accuracy looks strong and beats the single-human ceiling, but roughly half of it is a selection artefact; the defensible figure is a mean within-stratum AUROC of 0.592.
  • —emolia-dim is not a task these heads can do, and we report no number for it.
  • —Across both: ranking is decent, absolute calibration is not. Recalibrate on your own domain before thresholding.

Full results (all 61 dimensions)

Held-out validation, 10 % bucket-stratified split, taken after key de-duplication. bias = mean(prediction) − mean(target) on validation (caveat 2). Sorted by Pearson r.

#dimensionbandarchparamsn_trainn_valMAERMSEPearson rSpearman ρbiaslabel rangelabel mean
1durationstrongmlp2h966480,289266,78029,6420.3870.5860.9980.997+0.0253.0 … 30.013.344
2score_overall_qualitystrongmlp2h966480,289266,77929,6430.0880.1140.9490.941-0.0051.29 … 3.662.792
3Genderstrongmlp2h966480,289266,77929,6430.260.3730.940.907+0.036-2.666 … 2.291-0.096
4score_background_qualitystrongmlp2h966480,289266,78029,6420.1050.1410.9350.911+0.0031.523 … 4.5233.463
5score_content_enjoymentstrongmlp2h966480,289266,78029,6420.0730.0950.9330.906+0.0223.098 … 5.8834.877
6Monotone_vs._Expressivestrongmlp2h966480,289266,78029,6420.2140.2880.9010.888-0.0300.408 … 4.5272.402
7Serious_vs._Humorousstrongmlp1_h12898,561266,77929,6430.2660.3590.8960.876+0.032-0.274 … 5.0161.507
8High-Pitched_vs._Low-Pitchedstrongmlp2h966480,289266,78029,6420.1080.1470.8940.882+0.0110.544 … 4.4611.782
9Recording_Qualitystrongmlp1_h12898,561266,78129,6410.1820.2340.8930.896+0.0240.757 … 4.6562.471
10Amusementstrongmlp2h966480,289266,78029,6420.3380.4410.890.859+0.153-0.01 … 4.6720.865
11Concentrationstrongmlp1_h12898,561266,78129,6410.260.3440.8880.881+0.074-0.003 … 4.5620.862
12Vulnerable_vs._Emotionally_Detachedstrongmlp2h966480,289266,62129,6250.2290.3010.8490.837+0.041-0.049 … 4.0741.46
13Intereststrongmlp2h966480,289266,77929,6430.250.3510.8340.817-0.060-0.051 … 3.652.024
14Confident_vs._Hesitantstrongmlp2h966480,289266,78129,6410.2510.3430.8240.789+0.0890.085 … 4.1561.281
15Arousalstrongmlp2h966480,289266,78029,6420.2960.3950.8220.816+0.024-0.015 … 5.0191.897
16score_speech_qualitystrongmlp2h966480,289266,77929,6430.0450.0570.8070.721+0.0171.801 … 2.3611.924
17Embarrassmentstrongmlp2h643251,457266,78029,6420.2480.3290.7990.76+0.138-0.095 … 2.9880.392
18Submissive_vs._Dominantstrongmlp2h966480,289266,78129,6410.1740.2440.7950.77+0.047-0.656 … 2.3890.483
19Agestrongmlp2h966480,289266,78029,6420.1780.2740.7920.794-0.0360.077 … 5.4182.961
20Teasingstrongmlp2h643251,457266,78029,6420.3080.3790.7770.753+0.137-0.012 … 3.6620.548
21Soft_vs._Harshstrongmlp2h966480,289266,77929,6430.2230.3230.7540.721-0.062-1.956 … 1.7580.439
22Background_Noisestrongmlp1_h12898,561266,78229,6400.180.2340.7530.746-0.010-0.036 … 2.1310.549
23Contemplationstrongmlp2h966480,289266,78029,6420.3570.4830.7480.726+0.241-0.039 … 4.1840.504
24Angerstrongmlp2h966480,289266,78029,6420.4910.60.7280.516+0.388-0.02 … 6.1410.371
25talking_speedstrongmlp1_h12898,561266,78129,6412.5923.6830.7260.75-0.4720.124 … 100.69414.908
26Warm_vs._Coldstrongmlp1_h12898,561266,56229,6170.3140.4380.7240.699-0.001-2.215 … 2.7380.476
27Authenticitystrongmlp1_h12898,561266,78029,6420.1320.1690.7180.701+0.0261.594 … 4.1762.918
28Impatience_and_Irritabilitystrongmlp2h966480,289266,78029,6420.5630.6910.7180.598+0.419-0.075 … 5.7070.533
29Intoxication_Altered_States_of_Consciousnessstrongmlp2h966480,289266,78029,6420.3830.530.6960.61+0.279-0.111 … 4.5620.249
30Pleasure_Ecstasystrongmlp2h643251,457266,78029,6420.3330.4270.6960.706+0.181-0.014 … 4.7930.53
31Sexual_Luststrongmlp2h966480,289266,78029,6420.3530.4580.6830.637+0.250-0.083 … 4.2930.295
32Malevolence_Malicestrongmlp2h643251,457266,78129,6410.3860.4630.6790.466+0.314-0.042 … 3.6030.263
33Elationstrongmlp2h966480,289266,77929,6430.4530.5970.6670.676+0.277-0.024 … 5.5390.689
34Valencestrongmlp2h966480,289266,78029,6420.5880.8410.6450.624-0.176-4.555 … 4.1060.572
35Hope_Enthusiasm_Optimismstrongmlp2h643251,457266,78029,6420.5980.7770.640.611+0.357-0.014 … 6.1840.93
36Contemptstrongmlp2h643251,457266,77929,6430.410.4860.6380.519+0.325-0.013 … 4.7340.269
37Astonishment_Surprisestrongmlp2h643251,457266,78129,6410.5160.6610.6320.561+0.398-0.029 … 5.1330.446
38Contentmentstrongmlp2h643251,457266,77929,6430.3090.4040.630.655+0.1350.0 … 3.9610.551
39Bitternessstrongmlp2h966480,289266,78129,6410.3440.4020.6120.54+0.250-0.032 … 4.2540.204
40Longingstrongmlp2h643251,457266,77929,6430.3050.3740.6070.498+0.233-0.062 … 3.9670.18
41Doubtmediummlp2h643251,457266,77929,6430.3610.4230.5930.484+0.271-0.019 … 4.3910.236
42Confusionmediummlp2h643251,457266,78029,6420.4470.5450.5670.475+0.354-0.015 … 4.5270.273
43Affectionmediummlp2h643251,457266,77929,6430.5110.6390.550.465+0.383-0.023 … 5.2150.399
44Disgustmediummlp2h643251,457266,77929,6430.3990.4590.5430.39+0.313-0.026 … 3.5430.216
45Emotional_Numbnessmediummlp2h643251,457266,78029,6420.3080.4040.5430.397+0.242-0.021 … 4.0510.154
46Sournessmediummlp2h966480,289266,78029,6420.410.470.5280.389+0.316-0.031 … 3.9820.232
47Triumphmediummlp2h643251,457266,78029,6420.4770.5580.510.403+0.381-0.053 … 4.8320.217
48Sadnessmediummlp2h643251,457266,78029,6420.4330.5420.5030.345+0.372-0.01 … 4.8360.155
49Pridemediummlp2h966480,289266,77829,6440.3970.490.5020.522+0.280-0.009 … 4.6290.298
50Thankfulness_Gratitudemediummlp2h966480,289266,78029,6420.7620.9160.4950.411+0.652-0.056 … 4.8320.425
51Helplessnessmediummlp2h643251,457266,78029,6420.3830.4790.4850.557+0.321-0.019 … 3.7050.129
52Distressmediummlp2h643251,457266,78129,6410.5010.6260.4810.289+0.436-0.0 … 4.6990.159
53Fatigue_Exhaustionmediummlp2h643251,457266,78029,6420.3260.4410.4780.374+0.271-0.092 … 5.3790.104
54Disappointmentmediummlp2h966480,289266,77929,6430.5020.5950.4650.434+0.393-0.0 … 4.1520.289
55Infatuationmediummlp2h643251,457266,77929,6430.4550.5530.440.365+0.395-0.059 … 4.6760.128
56Jealousy_&_Envymediummlp2h966480,289266,78129,6410.3560.430.4250.356+0.293-0.052 … 4.0780.169
57Shamemediummlp2h643251,457266,78129,6410.4080.480.4080.36+0.348-0.023 … 5.0660.131
58Painmediummlp2h966480,289266,78029,6420.2140.3210.3920.199+0.178-0.02 … 5.070.046
59Fearmediummlp2h643251,457266,78029,6420.4050.4640.390.255+0.349-0.052 … 4.1370.094
60Reliefmediummlp1_h6449,281266,77829,6440.5940.7090.3550.317+0.491-0.057 … 5.1910.297
61Awemediummlp2h643251,457266,78029,6420.4610.5340.3380.235+0.413-0.026 … 4.7310.087

(Rows with `bias` above +0.35 are the ones caveat 2 is about. `Fear` +0.349 and `Confusion` +0.354 sit just under.)