besimple-ai/vocal-affect-bench

VocalAffectBench

VocalAffectBench is a test-only benchmark for evaluating whether AI audio models can identify expressed vocal emotion from raw audio.

Paper: VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models

The benchmark targets the expressed emotion — what the speaker conveys through vocal tone, prosody, pace, intensity, and pauses — not inferred internal state.

Contents

• —280 human-recorded English WAV clips, totalling 2.32 hours.
• —7 emotion classes, balanced at 40 clips each: angry, disgusted, fearful, happy, neutral, sad, surprised.
• —9 tracked baseline model outputs.
• —Audio recorded at 16 kHz mono.

Released files:

• —audio/*.wav — benchmark audio files, named by audio_id.
• —data/metadata.jsonl — per-clip metadata: emotion label, duration, sample rate.
• —data/audio-metadata.csv — flat CSV version of clip metadata.
• —data/predictions.csv — all baseline model predictions.
• —data/leaderboard-summary.csv — aggregate baseline leaderboard.
• —baselines/predictions/*.csv — per-model prediction files.
• —baselines/run-metadata/*.json — reproducibility metadata for new live runs.
• —baselines/results.csv — aggregate leaderboard (same as above).
• —scripts/ — model execution, scoring, validation, and report utilities.
• —paper/ — paper PDF and LaTeX source.

The public release does not include transcripts, script text, speaker identities, or demographic metadata.

Task

Each item is an audio clip of a person expressing one of the seven target emotions. Models are evaluated under a raw-audio-only protocol — the model receives only the audio file and a fixed prompt listing the allowed labels. No transcripts, speaker metadata, domain hints, or post-processing are permitted in the main setting.

The prompt instructs the model to:

Choose exactly one primary expressed emotion from the allowed label set. Base your answer only on the expressed vocal tone, prosody, pace, intensity, pauses, and wording. Do not infer the speaker's private internal state.

Label Set

angry  |  disgusted  |  fearful  |  happy  |  neutral  |  sad  |  surprised

Metrics

VocalAffectBench reports overall accuracy (correct / total scored clips). Because the benchmark is balanced at 40 clips per class, overall accuracy equals macro accuracy — no separate macro metric is needed.

Baselines

Current tracked baselines (9 models):

All baselines use the raw-audio-only protocol with no transcript-derived inference. Provider-native outputs are converted to exactly one benchmark label using the documented deterministic mappings.

modal_inkling, oruk_resonance_2, and gemini_3_8_flash are post-publication baselines. The paper is a frozen snapshot of the original release and is intentionally not regenerated when new baseline data is added.

Install and Reproduce

The release scripts require Node.js 20 or newer and have no third-party package dependencies:

npm install
npm test
npm run reproduce

npm run reproduce validates the balanced audio set and regenerates data/predictions.csv, data/leaderboard-summary.csv, per-model prediction files, and baselines/results.csv. It never changes anything under paper/.

To run the same Modal-hosted Thinking Machines Lab Inkling model used by the published modal_inkling baseline, set these values in the environment or in a private .env file:

MODAL_INKLING_ENDPOINT=https://<your-endpoint-host>
MODAL_PROXY_TOKEN_ID=wk-...
MODAL_PROXY_TOKEN_SECRET=ws-...

Then smoke-test one clip or run and publish the complete resumable baseline:

node scripts/run_model.cjs --model modal_inkling --limit 1

node scripts/run_model.cjs \
  --model modal_inkling \
  --concurrency 4 \
  --resume \
  --publish

The adapter accepts the Modal base URL, /v1 URL, or full /v1/chat/completions URL. It uses 16 kHz mono WAV input, text-before-audio message order, reasoning_effort=max, and no inference-time label correction. The .env file and resumable runs/ directory are ignored by Git.

To run Oruk Resonance 2, add its server-side API key to the same private .env file. The Oruk organization must also have Resonance 2 access approved:

ORUK_API_KEY=oruk_live_...

Smoke-test one clip, then run and publish the complete resumable baseline:

node scripts/run_model.cjs --model oruk_resonance_2 --limit 1

node scripts/run_model.cjs \
  --model oruk_resonance_2 \
  --concurrency 4 \
  --resume \
  --publish

The adapter calls POST /v1/audio/resonance-2?regime=f1 with raw audio only. Because Resonance 2 is multilabel while VocalAffectBench requires exactly one class, it selects the highest continuous score among the seven benchmark classes. Oruk's native scared score maps to fearful; the other six labels match directly. Stable per-clip request IDs make retries idempotent.

To run Gemini 3.8 Flash, provide a Gemini Developer API key in the private env file:

GEMINI_API_KEY=...

Then smoke-test one clip before starting the complete resumable run:

node scripts/run_model.cjs --model gemini_3_8_flash --limit 1

node scripts/run_model.cjs \
  --model gemini_3_8_flash \
  --concurrency 4 \
  --resume \
  --publish

The adapter sends each WAV inline to gemini-3.8-flash with the fixed benchmark prompt and a seven-label structured-output schema. It rejects responses unless Gemini's usage metadata confirms that audio tokens were processed.

Data Format

metadata.jsonl

Each line is a JSON object:

{
  "audio_id": "06Eg9dXO99fAAti4HB34",
  "file_name": "audio/06Eg9dXO99fAAti4HB34.wav",
  "required_emotion": "neutral",
  "duration_seconds": 12.4,
  "sample_rate": 16000,
  "channels": 1
}

predictions.csv

audio_id, required_emotion, model_name, provider_model, predicted_label, mapped_label, confidence, correct, error

• —predicted_label — raw label returned by the model.
• —mapped_label — normalised to the 7-class label set.
• —correct — true / false.

Use and Limits

VocalAffectBench is intended for diagnostic evaluation, provider comparison, and regression tracking of audio emotion models. It is not intended as a training corpus, hidden leaderboard, universal emotion-quality measure, or biometric dataset.

Emotion labels reflect the expressed vocal performance, not speaker demographics or inferred mental states.

Citation

@misc{vocalaffectbench2026,
  title  = {VocalAffectBench: Evaluating Vocal Emotion Recognition in AI Audio Models},
  author = {Debaupte, Luc and Baumgartner, Tyler and Tai, Brandon and Fan, Candice and Wang, Bill and Zhong, Yi},
  year   = {2026},
  note   = {Benchmark dataset}
}

License

The software and documentation are released under the MIT License. The audio recordings are subject to the additional Audio Data License in LICENSE and must not be used for voice cloning.