arcada-labs/event-bench

Event Bench

29-turn multi-turn speech-to-speech benchmark for evaluating voice AI models as an event planning assistant.

Part of Audio Arena, a suite of 6 benchmarks spanning 221 turns across different domains. Built by Arcada Labs.

Leaderboard | GitHub | All Benchmarks

Dataset Description

The model acts as an event planning assistant managing venue bookings, catering, and guest logistics. The conversation features cascading changes — a venue switch triggers catering repricing, a guest count update triggers capacity checks — along with mid-sentence self-corrections, retroactive date changes, and multi-request reversals that test whether the model can track compounding state changes.

What This Benchmark Tests

• —Tool use: 6 functions — venue booking, catering quotes, guest management, availability checks, and more
• —Cascading state changes: Venue switch triggers catering repricing, guest count changes trigger capacity rechecks
• —Mid-sentence self-corrections: Speaker changes details partway through a request
• —Vague pronoun resolution: Ambiguous references ("that one", "the other place") requiring context
• —Wrong-math correction: User provides incorrect arithmetic the model must not blindly accept
• —Multi-request reversals: Undoing multiple changes in a single turn
• —Ambiguous add-on disambiguation: Add-ons that could apply to multiple entities
• —Hypothetical reasoning: "What if" scenarios the model must handle without committing state
• —Phone number swap: Contact number correction mid-conversation
• —Retroactive date change: Changing the event date after downstream bookings are already set
• —False memory traps: 3 turns asserting things that never happened
• —Cross-entity state tracking: Keeping venue, catering, and guest details consistent

Dataset Structure

event-bench/
├── audio/                          # TTS-generated audio (1 WAV per turn)
│   ├── turn_000.wav
│   ├── turn_001.wav
│   └── ... (29 files)
├── real_audio/                     # Human-recorded audio
│   ├── person1/
│   │   └── turn_000.wav ... turn_028.wav
│   └── person2/
│       └── turn_000.wav ... turn_028.wav
├── benchmark/
│   ├── turns.json                  # Turn definitions with golden answers
│   ├── hard_turns.json             # Same as turns.json but input_text=null (audio-only)
│   ├── tool_schemas.json           # Tool/function schemas (6 tools)
│   └── knowledge_base.txt          # Event planning KB
└── metadata.jsonl                  # HF dataset viewer metadata

Metadata Fields

Audio Format

• —Format: WAV, 16-bit PCM, mono
• —TTS audio: Generated via text-to-speech
• —Real audio: Human-recorded by multiple speakers, same transcript content

Usage

With Audio Arena CLI

pip install audio-arena  # or: git clone + uv sync

# Run with a text model
uv run audio-arena run event_bench --model claude-sonnet-4-5 --service anthropic

# Run with a speech-to-speech model
uv run audio-arena run event_bench --model gpt-realtime --service openai-realtime

# Judge the results
uv run audio-arena judge runs/event_bench/<run_dir>

With Hugging Face Datasets

from datasets import load_dataset

ds = load_dataset("arcada-labs/event-bench")

Evaluation

Models are judged on up to 5 dimensions per turn:

For speech-to-speech models, a 6th turn_taking dimension evaluates audio timing correctness.

See the full methodology for details on two-phase evaluation, penalty absorption, and category-aware scoring.

Part of Audio Arena

Citation

@misc{audioarena2026,
  title={Audio Arena: Multi-Turn Speech-to-Speech Evaluation Benchmarks},
  author={Arcada Labs},
  year={2026},
  url={https://audioarena.ai}
}