manojkumarcs/indic-diarbench

Indic DiarBench

A multilingual joint diarization and ASR benchmark for Indian languages, spanning all 22 scheduled languages of India with approximately 108 hours of natural multi-speaker audio.

Dataset Summary

Indic DiarBench is a conversational speech benchmark designed to evaluate speaker-attributed ASR in realistic multi-speaker settings for Indian languages. All annotations are human-corrected with time-aligned, speaker-attributed transcriptions. The dataset captures conversational nuances prevalent in Indian speech, such as English code-mixing, dialectal variation, and frequent speaker overlap.

Recording Conditions

The corpus includes three acoustic conditions designed to capture diverse real-world scenarios:

Per-Language Statistics

Durations are in hours. NF = near-field, FF = far-field, ITW = in-the-wild.

Annotation Pipeline

All recordings are annotated using a unified human-in-the-loop pipeline:

1. 1.Bootstrap Transcription — Initial transcripts generated using multiple independent ASR systems, presented to annotators as editable drafts.
2. 2.Human Transcription & Speaker Attribution — Professional annotators produce time-aligned, speaker-attributed transcriptions. No machine-generated annotation is retained without human validation.
3. 3.Code-Mixed Transcription — Annotators produce two transcription formats: native-script (all text in Indic script) and normalized (English words in Roman script, numerals in Arabic digits).
4. 4.Quality Control — Dedicated quality checkers (2–3 per language) verify transcription consistency, code-mixing conventions, speaker timestamps, and labels. Overlapping speech segments require multiple review rounds.
5. 5.Expert Review — In-house language-specific experts perform final quality checks.

Dataset Fields

Identifying samples

sample_id is unique across the entire benchmark (all 1,164 samples), so it is the key to use when reporting or joining per-sample results.

recording_id exists because the 1,164 clips come from only 590 distinct source recordings — 47 recordings contribute more than one clip, and one contributes 21. Clips from the same recording are not independent samples: they share speakers, channel and acoustic conditions. Group by recording_id when splitting data or aggregating metrics, otherwise a handful of recordings will dominate the average.

recording_id is deliberately anonymised and carries no information about the source medium. Its form is <language>_<condition>_<nnn>, where condition is nf (near-field), ff (far-field) or itw (in-the-wild).

Usage

from datasets import load_dataset

# Load a specific language
ds = load_dataset("sarvam/indic-diarbench-new", "Hindi", split="test")
sample = ds[0]

print(f"Sample:    {sample['sample_id']}")
print(f"Recording: {sample['recording_id']}")
print(f"Language:  {sample['language']}")
print(f"Duration:  {sample['duration_seconds']:.1f}s")
print(f"Speakers:  {sample['num_speakers']}")
print(f"Condition: {sample['dataset_type']}")

# Access speaker-attributed transcript
for seg in sample['annotated_transcript'][:5]:
    print(f"  [{seg['start_time']:.1f}-{seg['end_time']:.1f}] {seg['speaker_id']}: {seg['transcript']}")

Aggregate per recording rather than per clip, so that recordings contributing many clips do not dominate:

import collections

by_recording = collections.defaultdict(list)
for s in ds:
    by_recording[s["recording_id"]].append(s["sample_id"])

print(f"{len(ds)} clips from {len(by_recording)} recordings")

Baseline Results

Duration-weighted aggregate metrics across all three acoustic conditions:

Relation to the original release

Derived from sarvamai/indic-diarbench. Audio is byte-identical and all annotations, timings and counts are unchanged; only the identifier and language columns differ.

1. 1.`language` is now the full language name everywhere. In the original, the 210 in-the-wild samples used ISO codes (bn, gu, hi, kn, ml, mr, or, pa, ta, te) while the 954 near- and far-field samples used full names, so language was not comparable across dataset_type.
2. 2.`file_name` replaced by `recording_id`. The original file_name held the source filename, which for in-the-wild samples was the YouTube video id; it was also duplicated in audio.path. Both are now anonymised, with audio.path set to <sample_id>.wav. recording_id retains the one piece of information that file_name carried and users need: which clips came from the same source recording.

Citation

@inproceedings{mehendale2026indicdiarbench,
  title={Indic DiarBench: A Multilingual Joint Diarization and ASR Benchmark for Indian Languages},
  author={Mehendale, Deovrat and Mehndiratta, Aditya and Rathi, Dhruv and Bhogale, Kaushal and Khapra, Mitesh M.},
  year={2026}
}

License

This dataset is released under the CC BY 4.0 license.