modulate/entity-transcription-benchmark

Entity Transcription Benchmark

Measures whether a speech recognition system transcribes named entities correctly — as distinct from word error rate.

WER weights every token equally. The tokens that matter for redaction, lookup, routing and search are proper nouns, and they are a small fraction of any transcript. A system can improve WER while getting worse at exactly the words a downstream consumer needs, and nothing in the standard evaluation will show it.

2,151 clips, 6.0 hours, 3,245 entity annotations across two acoustically and linguistically different domains.

Built by Chris James and Angel Carrillo-Bermejo at Modulate.

CC-BY-SA-4.0. See Licensing and attribution for upstream sources and terms.

Quick start

from datasets import load_dataset
ds = load_dataset("modulate/entity-transcription-benchmark", split="test")
print(ds[0])

Or benchmark a system end to end. The harness transcribes every clip through your endpoint and scores the result — no transcript file needed up front:

huggingface-cli download modulate/entity-transcription-benchmark \
  --repo-type dataset --include "scripts/*" --local-dir .
cd scripts && pip install -r requirements.txt

export MY_STT_KEY=...
python run_benchmark.py --endpoint https://api.example.com/v1/transcribe \
  --api-key-env MY_STT_KEY --response-path result.transcript \
  --system-name my-system --out runs/mine.jsonl

python score_benchmark.py --hyps runs/mine.jsonl

Full instructions, vendor shortcuts and the scoring protocol are in Reproducing our numbers below.

Use the shipped scorer. Entity accuracy is highly sensitive to normalization — casing, punctuation, spoken numbers — and independently written scorers do not produce comparable numbers. matcher_v5 is the definition of correctness here.

Composition

MLCommons_TestPortion is spontaneous multi-speaker audio from US municipal and legislative meetings. References are verbatim lowercase.

Belebele_Fleurs_TestPortion is volunteers reading encyclopedic prose. References are cased.

Which subset to report

Report MLCommons as your headline number. It is spontaneous, acoustically harder, and independent of the public benchmark suites that models are commonly evaluated on and sometimes trained on.

Belebele and FLEURS are widely used, so training exposure cannot be ruled out for any given model. That subset is provided as a diagnostic: a large gap between a model's score on the two subsets is itself informative.

This is not hypothetical. In our own evaluation, one system led a second by 12.7 points on Belebele/FLEURS and by only 1.9 points on MLCommons — the same two systems, the same scorer, the same run. A large lead on the public-benchmark subset that shrinks to near-nothing on independent audio is the pattern this dataset exists to expose. We found it in our own system first.

No train split

This is an evaluation set. It ships as a single test split and is not partitioned for training. Nothing stops you training on it; doing so simply invalidates it as a measurement, for you and for anyone comparing against you.

Entity types

GPE is polities. LOC is physical places that are not polities. NORP is nationality, religious and political groups including adjectives. OTHER_NAMED covers named events, works, products, laws and facilities, merged because two independent annotation passes could not separate those classes reliably — see `ANNOTATION_GUIDELINES.md` §5.

Metric and difficulty tiers

Per-entity accuracy: the fraction of annotated spans for which the hypothesis contains a match under matcher_v5 normalization.

Report tier A only for the headline figure, and give `n`.

Tier B is 33.5% of MLCommons and 22.6% of Belebele/FLEURS, because meeting speech is full of bare first names. Any English recognizer transcribes david and jim correctly, so including them inflates the score without discriminating between systems. They are retained so users can re-tier or report both; a tier A+B figure is not comparable to a tier A one.

Report as: entity accuracy, MLCommons, tier A, n=1086.

Fields

Every span is an exact substring of its transcript, verified on every build. 98.8% additionally carry character offsets; the remainder are cases where one string was annotated more times than it occurs in the transcript.

Case follows the source. MLCommons references are verbatim lowercase, so epa, boston red sox and clean air act are correct as written. The scorer normalizes case and punctuation, so a system that capitalizes properly is not penalized.

Annotation quality

The spans and types in this dataset were produced automatically, not written by human annotators. This is the most important thing to know about it, and we would rather state it plainly than have it discovered.

Types. 2,813 of 3,245 spans are a documented one-to-one rename of the source corpus's own labels, with no model judgement involved. The remaining 522 carried source labels that were not entity types at all — a class containing real entities alongside things that were not names. Those were classified by two independent model passes over a closed inventory:

• —raw agreement 82.5%, Cohen's κ = 0.785
• —351 auto-decided where both passes agreed
• —69 disagreements adjudicated by hand
• —90 spans judged not to be entities and removed from the release

Spans. Recovered by LLM extraction after a capitalization heuristic returned zero spans on the lowercase MLCommons references, then corrected by a deterministic boundary sweep implementing the guideline's §4 rules. Every edit was re-validated as an exact substring and reverted if it broke the contract.

Review. A stratified sample of 100 spans was reviewed by hand against `ANNOTATION_GUIDELINES.md` §8. 100 were judged correct, giving a span accuracy of 100% (95% Wilson CI 96.3–100.0).

Caveat on that figure: the review was conducted by one reviewer, who is also an author of the dataset. Single-reviewer self-assessment is the weakest form of validation and the interval above does not account for reviewer bias. The full review sheet is published as span_review.csv so the judgements can be audited or disputed, and we would welcome an independent pass. Treat the true error rate as somewhere at or above zero and below the upper bound of the interval.

Substring validation proves a span exists in its reference. It does not prove the span is correct. Those are different claims and are reported separately above.

Working papers included

Published in the repository so the decisions can be checked rather than trusted:

Duplicates

FLEURS has multiple speakers read the same sentence. 683 clips repeated a transcript that already appeared elsewhere, which would have silently weighted those entities. They were collapsed to one clip per unique transcript, keeping the best member by alignment status, then annotation count, then duration. Nearly all removals were from the Belebele/FLEURS subset (1,469 → 797); MLCommons lost 11.

The published set contains no duplicate transcripts, exact or near (Jaccard ≥ 0.8 over word 5-grams).

Known limitations

• —Annotation quality is machine-generated with single-reviewer validation, as described above.
• —47 clips carry zero entity annotations after non-entity removal. They are retained deliberately as negative examples: a system should predict nothing on them. Exclude them if your metric cannot handle empty references.
• —Six entity types. No dates, money, times or quantities.
• —English only.
• —Two domains: read encyclopedic prose, and US municipal and legislative meetings. Do not generalize to conversational, telephony, broadcast, or accented-L2 speech.
• —MLCommons references are verbatim transcripts and carry disfluencies and transcription noise from the source corpus.
• —Boundary artifacts remain in an unmeasured portion of unreviewed spans.

Authors

• —Chris James, Modulate — corpus construction and data collection
• —Angel Carrillo-Bermejo, Modulate — annotation pipeline, scoring, release

Licensing and attribution

Derived from three openly licensed corpora, audio redistributed unmodified:

• —MLCommons People's Speech — CC-BY-SA-4.0 — dataset
• —Belebele — CC-BY-SA-4.0 — dataset
• —FLEURS — CC-BY-4.0 — dataset

Share-alike propagates; this dataset is CC-BY-SA-4.0. Original contributions are the spans, types, tiers, offsets, guidelines and scorer.

Personal and sensitive information

The audio contains identifiable speakers. The MLCommons subset consists of recordings of public municipal and legislative meetings in which named individuals speak and are named by others; those names appear in the reference transcripts and are annotated as entities.

All audio is redistributed from openly licensed corpora under their original terms. Requests concerning a specific recording should be directed to the upstream corpus maintainer as well as to us.

Removal requests. If you appear in a recording in this dataset and want it removed, email ml.modulate@modulate.ai with the clip filename or a description of the recording. We will acknowledge within five working days and, where the request is valid, remove the clip and its annotations in the next version, listing the removal in the changelog without identifying the requester. Because the underlying audio is redistributed from upstream corpora, we will also point you to the relevant maintainer, since removal here does not remove it there.

We have added no information that makes any speaker more identifiable than they already are in the source corpora. No speaker identifiers, demographics or linkage data are published.

Reproducing our numbers

This repository ships the scoring harness, not just the data. It is vendor-neutral: point it at any speech-to-text endpoint, name the environment variable holding your key, and say where the transcript sits in the JSON response.

huggingface-cli download modulate/entity-transcription-benchmark \
  --repo-type dataset --include "scripts/*" --local-dir .
cd scripts && pip install -r requirements.txt

export MY_STT_KEY=...
python run_benchmark.py \
  --endpoint https://api.example.com/v1/transcribe \
  --api-key-env MY_STT_KEY \
  --response-path result.transcript \
  --system-name my-system \
  --out runs/mine.jsonl

python score_benchmark.py --hyps runs/mine.jsonl

Convenience adapters for Deepgram, OpenAI and ElevenLabs are available through --provider, so you do not have to look up their response shapes. They are shortcuts; the generic path above is the supported one. Endpoints requiring no authentication work by omitting --api-key-env.

Collection is resumable, so an interrupted run continues where it stopped. API keys are read from the environment and never from the command line, so they stay out of shell history and out of the run manifest.

The scoring protocol

Five things determine the number. Change any of them and the result is no longer comparable to ours:

entity_tiers is an array parallel to entities, so a single clip can hold both tier A and tier B spans; the scorer filters spans, not clips. See Metric and difficulty tiers for what each tier contains and why the headline uses tier A.

matcher_v5 does considerably more than substring matching: Unicode canonicalization, abbreviation aliases, spoken-form reconstruction for numbers, years and dates, and contiguous token matching. A scorer written from scratch will not produce comparable figures, which is why score_benchmark.py refuses to run when the matcher is missing or its self-test fails.

Hypotheses are scored exactly as the endpoint returns them. If your system has a formatting layer, benchmark it on and off and state which one your headline used.

Two conventions apply to the figures we publish, which the scorer does not apply for you: clips where both systems returned nothing are excluded, and a category is not reported within a subset unless it holds enough examples to be meaningful. Run the scorer straight and it counts every span.

Reporting

Scoring writes a per-span hits CSV, a summary JSON and a run manifest. Publish the summary and the manifest together — a percentage without the manifest is not reproducible by anyone, including us. Pin the annotations you scored against with --revision <commit-sha>, and record the vendor model string and run date, since hosted models change without version bumps.

If your number disagrees with ours, diff at the span level rather than arguing about the percentage:

python score_benchmark.py --hyps runs/mine.jsonl --compare reference/<run>.hits.csv

The full protocol, every option and the known limits are in scripts/BENCHMARK.md.

Versioning

Current version: 1.0.

Corrections are expected and ship as subsequent versions. Always report the version you evaluated against — an entity accuracy figure is not comparable across versions of the ground truth.

Feedback

If you find a wrong span, a wrong type, or a tier assignment you disagree with, open a discussion on this repository. Disputed annotations are more useful to us than silence, and corrections are credited in the changelog.

Direct contact: ml.modulate@modulate.ai — reaches the Modulate ML team.

Citation

@misc{entity_transcription_benchmark_2026,
  title        = {Entity Transcription Benchmark},
  author       = {James, Chris and Carrillo-Bermejo, Angel},
  year         = {2026},
  publisher    = {Modulate},
  howpublished = {\url{https://huggingface.co/datasets/modulate/entity-transcription-benchmark}}
}

Please also cite the upstream corpora: MLCommons People's Speech, Belebele, and FLEURS.