google/red_ace_asr_error_detection_and_correction
RED-ACE Dataset Summary This dataset can be used to train and evaluate ASR Error Detection or Correction models. It was introduced in the RED-ACE paper (Gekhman et al, 2022). The dataset contains ASR outputs on the LibriSpeech corpus (Panayotov et al., 2015) with annotated transcription errors. Dataset Details The LibriSpeech corpus was decoded using Google Cloud Speech-to-Text API, with the default and video models. The word-level confidence was… See the full description on the dataset page: https://huggingface.co/datasets/google/red_ace_asr_error_detection_and_correction.
RED-ACE
Dataset Summary
This dataset can be used to train and evaluate ASR Error Detection or Correction models. It was introduced in the RED-ACE paper (Gekhman et al, 2022).
The dataset contains ASR outputs on the LibriSpeech corpus (Panayotov et al., 2015) with annotated transcription errors.
Dataset Details
The LibriSpeech corpus was decoded using Google Cloud Speech-to-Text API, with the default and video models. The word-level confidence was enabled and is provided as part of the transcription hypothesis. To annotate word-level errors (for the error detection task), the hypothesis words were aligned with the reference (correct) transcription to find an edit path (insertions, deletions and substitutions) with the minimum edit distance (from the hypothesis to the reference). The hypothesis words with deletions and substitutions were then labeled as ERROR (1), the rest were labeled as NOTERROR (0).
Data format
The dataset has train, developement and test splits which correspond to the splits in Librispeech.
The data contains json lines with the following keys (note that asrhypothesis[i], confidencescores[i] and error_labels[i] correpond to the same word):
"id"- The librispeech id."truth"- The reference (correct) transcript from Librispeech."asr_model"- The ASR model used for transcription."librispeech_pool": Corresponds to the original pool (split) in the librispeech data."asr_hypothesis"- The transcription hypothesis."confidence_scores"- The word-level confidence scores provided as part of the transcription hypothesis."error_labels"- The error labels (1 error, 0 not error) that were obtained by alighning the hypothesis and the reference.
Here is an example of a single data item:
{
"id": "test-other/6070/86744/6070-86744-0024",
"truth": "my dear franz replied albert when upon receipt of my letter you found the necessity of asking the count's assistance you promptly went to him saying my friend albert de morcerf is in danger help me to deliver him",
"asr_model": "default",
"librispeech_pool": "other",
"asr_hypothesis": ["my", "dear", "friends", "replied", "Albert", "received", "my", "letter", "you", "found", "the", "necessity", "of", "asking", "the", "county", "assistance", "you", "promptly", "went", "to", "him", "saying", "my", "friend", "all", "but", "the", "most", "stuff", "is", "in", "danger", "help", "me", "to", "deliver", "it"],
"confidence_scores": ["0.9876290559768677", "0.9875272512435913", "0.6921446323394775", "0.9613730311393738", "0.9413103461265564", "0.6563355922698975", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "1.0", "1.0", "1.0", "1.0", "1.0", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.9876290559768677", "0.5291957855224609", "0.5291957855224609"],
"error_labels": ["0", "0", "1", "0", "0", "1", "0", "0", "0", "0", "0", "0", "0", "0", "0", "1", "0", "0", "0", "0", "0", "0", "0", "0", "0", "1", "1", "1", "1", "1", "0", "0", "0", "0", "0", "0", "0", "1"]
}Loading the dataset
The following code loads the dataset and locates the example data item from above:
from datasets import load_dataset
red_ace_data = load_dataset("google/red_ace_asr_error_detection_and_correction", split='test')
for example in red_ace_data:
if example['id'] == 'test-other/6070/86744/6070-86744-0024':
break
print(example) Citation
If you use this dataset for a research publication, please cite the RED-ACE paper (using the bibtex entry below), as well as the Librispeech paper mentioned above.
@inproceedings{gekhman-etal-2022-red,
title = "{RED}-{ACE}: Robust Error Detection for {ASR} using Confidence Embeddings",
author = "Gekhman, Zorik and
Zverinski, Dina and
Mallinson, Jonathan and
Beryozkin, Genady",
booktitle = "Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing",
month = dec,
year = "2022",
address = "Abu Dhabi, United Arab Emirates",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2022.emnlp-main.180",
doi = "10.18653/v1/2022.emnlp-main.180",
pages = "2800--2808",
abstract = "ASR Error Detection (AED) models aim to post-process the output of Automatic Speech Recognition (ASR) systems, in order to detect transcription errors. Modern approaches usually use text-based input, comprised solely of the ASR transcription hypothesis, disregarding additional signals from the ASR model. Instead, we utilize the ASR system{'}s word-level confidence scores for improving AED performance. Specifically, we add an ASR Confidence Embedding (ACE) layer to the AED model{'}s encoder, allowing us to jointly encode the confidence scores and the transcribed text into a contextualized representation. Our experiments show the benefits of ASR confidence scores for AED, their complementary effect over the textual signal, as well as the effectiveness and robustness of ACE for combining these signals. To foster further research, we publish a novel AED dataset consisting of ASR outputs on the LibriSpeech corpus with annotated transcription errors.",
}