puneetUMD/DSB-IFEval

DuplexSpeechBench–IFEval (DSB-IFEval)

Evaluating implicit instruction following in full-duplex voice agents.

⚠️ Preprint — under review. Please cite it as a preprint (see below).

DSB-IFEval tests whether a real-time voice agent can infer the turn-taking behavior a role implies — and execute it at the right moment on the conversational floor. It contains 1,038 evaluation cases built from 240 fixed user-side spoken interactions (8 assistant roles × 6 conversational probes × 5 instances), each presented under five conditioning protocols.

• —🌐 Project page: http://dsb-ifeval.github.io/
• —📄 Paper: https://arxiv.org/pdf/2609.03423

Conditioning protocols

Only the model's prompt changes across levels; the user audio is fixed.

Contents

spec/roles.yaml, probes.yaml, instructions.yaml   # roles, personas, probes, conditioning
data/manifest.jsonl   # 1,038 cases: expected action, verifier, trigger window, segments, audio paths
data/scripts.jsonl    # 245 user-side scripts (segment text + word timings)
data/cases.jsonl      # case definitions (system prompt per level, l4_expected)
audio/<audio_key>/turn1.wav, turn2.wav, meta.json # 24 kHz mono user audio + segment timeline

manifest.jsonl — one JSON object per case

Key fields: test_case_id, level (L0–L4b), role_id/role_name, probe_id, expected_action (LISTEN, TAKETURN, BACKCHANNEL, NOBACKCHANNEL, INTERRUPT, YIELD, CONTINUE, READBACK, ACCEPTOVERLAP), `l4expected (directive/persona), systeminstruction`, `verifier` (name + thresholds), `audio` (paths, speaker, sr, durations), `turns`, `triggerwindow, groundtruthtimestamps, segments (speech/silence with word-level timings), readback_target`.

Loading

import json
cases = [json.loads(l) for l in open("data/manifest.jsonl")]
import soundfile as sf
c = cases[0]
user_t1, sr = sf.read(f"{c['audio']['turn1_wav']}")   # 24 kHz mono

Evaluation

Two primary metrics, reported separately:

• —IAS — Instruction Adherence Score (deterministic). A per-action verifier checks whether the model performs the expected floor action within the injected trigger window.
• —PAS — Persona Adherence Score (LLM-judged, 0–100). Whether the spoken response fits the role in register and content, independent of timing.

Derived / additional metrics:

• —Entailment Gap = IAS(L1) − IAS(L2) — the cost of inferring the rule from a persona rather than receiving it explicitly (the headline metric).
• —Redundancy Gain = IAS(L3) − IAS(L2) — whether restating the persona-implied rule improves execution.
• —Role Tax = IAS(L3) − IAS(L1) — whether adding a persona degrades an already-explicit instruction.
• —Conflict Profile (L4) — distribution over {directive-wins, persona-wins, balanced, incoherent}.
• —SafetyOverride — on L4b safety conflicts, the fraction resolved persona-wins (safety-preserving).

Scoring code (verifiers, judge prompts, per-model adapters) is released together with the benchmark.

Provenance & license

User speech is synthesized with Coqui XTTS-v2 and force-aligned (torchaudio MMS_FA); pauses, chunk gaps, and barge-in timing are injected programmatically. This dataset is released under CC-BY-NC-4.0 for research use. Roles include safety-sensitive scenarios (medical triage, emergency dispatch) solely to evaluate conversational-policy behavior — not clinical or emergency decision-making.

Citation

@article{mathur2026duplexspeechbench,
  title={DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents},
  author={Mathur, Puneet and Manocha, Dinesh},
  journal={arXiv preprint arXiv:2609.03423},
  year={2026}
}