Datasetpublic

lilonghao/MM-ContextASR-Bench

MM-ContextASR Bench Metadata and evaluation splits for Multimodal Conversational Context for LLM-Based ASR: Data Construction, Training, and Benchmark. Dataset summary Config Examples Audio Context Primary metric mm_contextasr 1,250 (250 current utterances × 5 histories) 1,439 WAV files included Controlled user-assistant dialogue entity Recall kespeech 19,212 Source ID only Same-speaker speech and transcript CER, SER, entity Recall cv_yue 3,525… See the full description on the dataset page: https://huggingface.co/datasets/lilonghao/MM-ContextASR-Bench.

sourceHugging Faceotherupdated 5d agoView on Hugging Face

1likes169downloads

Dataset Card

MM-ContextASR Bench

Metadata and evaluation splits for Multimodal Conversational Context for LLM-Based ASR: Data Construction, Training, and Benchmark.

Dataset summary

Config	Examples	Audio	Context	Primary metric
`mm_contextasr`	1,250 (250 current utterances × 5 histories)	1,439 WAV files included	Controlled user-assistant dialogue	entity Recall
`kespeech`	19,212	Source ID only	Same-speaker speech and transcript	CER, SER, entity Recall
`cv_yue`	3,525	Source ID only	Same-speaker speech and transcript	CER, SER, entity Recall
`alimeeting`	2,850	Source segment ID and timing only	Same-speaker far-field enrollment	target-only CER, SER

The companion MM-ContextASR repository contains validation, four-mode input rendering, and scoring tools.

The three external-corpus configurations contain metadata and source-audio identifiers only; obtain their audio from the corresponding official releases.

MM-ContextASR scenarios

Every aligned group fixes current speech, reference transcription, and target entity while changing the preceding dialogue.

Scenario	Historical evidence	Diagnostic purpose
Irrelevant	Unrelated topic	Ignore distractors
Implicit	Related topic without the entity	Use indirect semantic cues
Explicit	Correct entity appears	Use direct contextual evidence
Correction	Historical ASR error corrected by assistant	Recover from transcript errors
Repeated Error	Historical error repeated by assistant	Resist error propagation

Record format

Each JSONL record contains a stable id, source dataset and split, task, language, current audio ID, current reference, ordered history turns, and source provenance. Task-specific fields include scenario/entity labels, accent, speaker/meeting IDs, timestamps, and overlap statistics.

MM-ContextASR records use repository-relative paths under audio/. Audio fields in the three external evaluation tracks are identifiers only. No internal object-store URI, signed URL, or machine-specific path is included.

json

{
  "id": "common_voice_yue_31189594__common_voice_yue_31189595",
  "dataset": "cv_yue",
  "split": "test",
  "task": "asr",
  "language": "yue",
  "current_audio_id": "common_voice_yue_31189595.mp3",
  "current_transcript": "官僚兼地主",
  "history": [
    {
      "role": "user",
      "audio_id": "common_voice_yue_31189594.mp3",
      "text": "睇我几有礼貌，去之前讲返声"
    }
  ],
  "speaker_id": "cv_yue_spk_0210a731f5caa5d4",
  "normalization": "OpenCC-t2s",
  "entities": [],
  "source": {
    "corpus": "Mozilla Common Voice",
    "release": "26.0 (2026-06-12)",
    "locale": "yue",
    "license": "CC0-1.0"
  }
}

Context modes

The canonical row contains all available history. The toolkit renders:

Mode	Historical user speech	Historical user text	Assistant text
No Context	✗	✗	✗
Text-only	✗	✓	✓
Speech-only	✓	✗	✓
Speech+Text	✓	✓	✓

Current-turn references, entity labels, and scenario labels are evaluation-only and must never be included in prompts.

Source-audio mapping

The external-corpus JSONL files preserve source identifiers but do not redistribute the corresponding audio. Resolve each record as follows:

Configuration	Mapping rule	Speaker relation
`kespeech`	Match the basename in `current_audio_id` and `history[].audio_id` against the official KeSpeech release. For example, locate `1000043_6674a7fd.wav` recursively under the corpus audio root; released basenames are unique within this evaluation set.	Both files share the row-level `speaker_id`; the speaker prefix in each filename provides an additional check.
`cv_yue`	Download Common Voice Cantonese 26.0 and resolve each ID as `clips/<audio_id>`, for example `clips/common_voice_yue_31189595.mp3`.	The privacy-preserving row-level `speaker_id` is shared by the current and history clips. It is stable within this release but is not the original Common Voice `client_id`.
`alimeeting`	Resolve the raw SLR119 recording as `Eval_Ali_far/audio_dir/<meeting_id>_<far_channel>.wav`, then extract `[current_start, current_end]` and `[history_start, history_end]`. The two derived clips intentionally share a basename but come from `current_far/` and `reference_far/`; `current_audio_id` is not a raw SLR119 filename.	Both intervals belong to the row-level target `speaker_id`. The history interval is a same-speaker enrollment/reference segment and is not necessarily earlier than the current interval.

For mm_contextasr, audio is included directly: current_audio and history[].audio are repository-relative paths under audio/current/ and audio/history/. In every external-corpus configuration, speaker_id denotes the identity shared by the current utterance and its user-speech context; assistant turns do not carry a speaker identity.

Evaluation

Use corpus-level CER and utterance-level SER after Chinese punctuation and whitespace normalization. CV-Yue additionally applies OpenCC t2s to references and predictions. AliMeeting references contain target-speaker words only.

For MM-ContextASR, report exact entity Recall per scenario and overall. The five examples in a group share current audio and are paired observations.

For CV-Yue entity Recall, use the provided 395 proper-name and domain-term spans covering 327 utterances. The annotations were extracted from references and audited under fixed span and type rules; they form a targeted diagnostic set rather than exhaustive entity annotation.

Licensing and citation

See LICENSES.md for per-config terms. Cite the paper and each source corpus used in your evaluation. The final paper citation will be added when an identifier is available.