keylazy/slurp-ear-masked-eval

SLURP: Semantic-Acoustic Masked Evaluation Dataset (EAR Metric)

Dataset Description

This dataset is a custom evaluation subset derived from the SLURP (Spoken Language Understanding Resource Package) dataset.

It is specifically engineered to evaluate the EAR (Execution and Repair) metric for active voice assistants. Using a forced-alignment masking protocol, real human audio is mathematically perturbed with high-amplitude white noise to create two strictly controlled experimental conditions: Answerable (Semantic-Invariant) and Unanswerable (Semantic-Degrading).

Dataset Structure

Data Fields

• —`slurp_id` (int64): The unique identifier corresponding to the original SLURP dataset instance.
• —`sentence` (string): The clean, ground-truth text transcript of the spoken command (e.g., "Schedule a meeting with John for 3 PM").
• —`annotation` (string): The original SLURP bracketed entity annotation.
• —`intent` (int64): The integer class label representing the specific SLURP intent.
• —`critical_word` (string): The execution-critical entity extracted from the annotation. If this word is removed, the command becomes underspecified.
• —`non_critical_word` (string): A semantic-invariant stopword or filler word extracted from the sentence (e.g., "a", "the", "please").

Audio Features (16kHz)

The dataset provides two paired audio features for every command, generated using precise time-alignment via Whisper:

• —`answerable_audio` (audio): The Semantic-Invariant variant. White noise has been injected over the exact millisecond timestamps of the non_critical_word. The acoustic surface is damaged, but the core command remains fully executable. This is used to test the model's Task Competence (C).
• —`unanswerable_audio` (audio): The Semantic-Degrading variant. White noise has been injected over the exact millisecond timestamps of the critical_word. The command is now intrinsically underspecified. This is used to test the model's Repair Behavior (R).