projecte-aina/corts_valencianes_asr_a

Dataset Card for Corts Valencianes - Speech Corpus of Valencian Parliamentary Sessions

The Corts Valencianes Speech Corpus is a rich dataset composed of speech recordings from the sessions of the Corts Valencianes. The corpus includes both clean and other quality segments, divided into short segments (less than 30 seconds) and long segments (more than 30 seconds). The total dataset encompasses 270 hours, 5 minutes, and 34 seconds of speech, including 239h 05m 24s for the short segments and 31h 00m 11s for the long segments, with a total of 2,621,096 words.

Table of Contents

• —Dataset Details
• —Dataset Description
• —Dataset Sources
• —Uses
• —Dataset Structure
• —Data Instances
• —Data Fields
• —Data Splits
• —Example Usage
• —Dataset Creation
• —Curation Rationale
• —Source Data
• —Data Collection and Processing
• —Who are the Source Data Producers?
• —Annotations
• —Personal and Sensitive Information
• —Citation
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations

Dataset Details

Dataset Description

This is the first version of the Corts Valencianes speech corpus for Valencian: a collection of speech recordings with transcriptions intended for Automatic Speech Recognition (ASR) applications. In recent years, data in Catalan language has increased considerably. However, most of the content corresponds to the central variant, while it is extremely scarce in Valencian. With this release we develop a speech corpus in Valencian, which will be very valuable mainly for training and evaluating speech recognition systems.

We used the content of the Corts Valencianes sessions: the audio segments were extracted from recordings the Valencian Parliament (Corts Valencianes) plenary sessions. Taking advantage of the manual transcriptions, we created high quality audio segments in Valencian along with the aligned transcriptions. The extensive time span covered by the sessions, from June 25, 2007, to September 21, 2023, provides a broad range of linguistic phenomena and topics, further enriching the corpus. With 433 sessions in total, the corpus is substantial and should provide ample data for various research and development purposes in speech recognition. The final corpus has been extracted March 5, 2024.

• —Curated by: Language Technologies Unit at the Barcelona Supercomputing Center (langtech@bsc.es). The data have been collected thanks to the intervention of the NEL-VIVES campaign, an initiative developed by Cenid, the Digital Intelligence Center of the University of Alicante.
• —Funded by: This work is funded by the Ministerio para la Transformación Digital y de la Función Pública - Funded by EU – NextGenerationEU within the framework of the project ILENIA with reference 2022/TL22/00215337 y 2022/TL22/00215334
• —Shared by: [More Information Needed]
• —Language(s) (NLP): ca (valencian)
• —License: CC-BY-NC-4.0

Dataset Sources

• —Repository: [More Information Needed]
• —Paper: [More Information Needed]

Uses

The purpose of this dataset is mainly for training automatic speech recognition (ASR) models in Valencian.

Dataset Structure

Data Instances

Each instance have the following structure:

DatasetDict({
    clean_train_short: Dataset({
        features: ['identifier','audio','segment_path','text'],
        num_rows: 46219
    })

Each data point is structured as:

• —Audio ID

>>data['clean_train_short'][0]['audio_id']
245_5_20201117_1041_0_8637_9300_155.68_159.20000000000002

• —Audio

>>data['clean_train_short'][0]['audio']
{'path': '/Users/sarahsolito/.cache/huggingface/datasets/downloads/extracted/9f760c175adf0af8127242f9468e48120f7682b20cf5c5813bfe481a108524bf/corts/corpus/speech/245_5_20201117_1041/245_5_20201117_1041_0_8637_9300_155.68_159.20000000000002.wav', 'array': array([-1.07421875e-02, -1.33972168e-02, -1.62353516e-02, ...,
        1.64794922e-03,  3.05175781e-05, -4.02832031e-03]), 'sampling_rate': 16000}

• —Relative Path

>>data['clean_train_short'][0]['relative_path']
corts/corpus/speech/245_5_20201117_1041/245_5_20201117_1041_0_8637_9300_155.68_159.20000000000002.wav

• —Transcription

>>data['clean_train_short'][0]['text'])
i açò és el que passa que estan parlant de coses que no estan enlloc

Data Fields

• —"identifier" : (string) → the unique audio identificator
• —"segment_path": (string) → the path to the audio
• —"start": (string) →the start timestamps of the audio
• —"audio": datasets.Audio(sampling_rate=16000) → the decoded audio array, and the sampling rate.
• —"text": (string) → clean version of the transcription

Data Splits

The dataset consists of a train, dev and test splits, for both short and long segments. The stat details are as follows:

Example Usage

To load the dataset:

from datasets import load_dataset
data = load_dataset("projecte-aina/corts_valencianes_asr_a",trust_remote_code=True)

To load the dataset with streaming enabled (recommended for large audio files):

from datasets import load_dataset
data = load_dataset("projecte-aina/corts_valencianes_asr_a",trust_remote_code=True, streaming=True)

Dataset Creation

Curation Rationale

The directory called "speech" contains all the speech files of the corpus, where "clean" and "other" for both short and long audios can be found.

Source Data

The content belongs to the Corts Valencianes and the data is released conforming their terms of use.

The data have been collected thanks to the intervention of the NEL-VIVES campaign, an initiative developed by Cenid, the Digital Intelligence Center of the University of Alicante.

Data Collection and Processing

The dataset's transcriptions are released in a clean version. The clean versions have been normalized at an orthographic level in lower-case. The normalization process was performed removing punctuation marks and characters that are not present in the Catalan alphabet. Number expansion was also perfomed.

In order to obtain a corpus of the highest possible quality, we also apply automatic language detection processes to each segment to prevent code-switching, and evaluate the quality of the transcriptions to eliminate both low quality segments and those that are not in Catalan.

Who are the source data producers?

The content belongs to the Corts Valencianes and the data is released conforming their terms of use.

Annotations

The dataset doesn't contain any additional annotation.

Personal and Sensitive Information

The dataset consists of Corts Valenciances parliamentary speeches and their transcription. The dataset contains no personal information except for speech, which is considered personal data. Consequently, the speakers' voices in this corpus have been subjected to anonymization treatment in compliance with applicable regulations, such as the General Data Protection Regulation (GDPR) in the European Union. You agree to not attempt to determine the identity of speakers in this dataset.

Citation

@misc{bscib32024,
      title={Corts Valencianes - Speech Corpus for Valencian ASR}, 
      author={Baybars, Kulebi},
      publisher={Barcelona Supercomputing Center},
      year={2024},
      url={},
      }

Considerations for Using the Data

Social Impact of Dataset

Cortes Valencianes is a source of speech data that will be valuable in development of speech technologies for Valencian.

Discussion of Biases

The language is limited to the parlamentary sessions used to create the corpus and may not be representative to all domains.

Other Known Limitations

Speakers, their gender and age are not identified and one or more speakers could be speaking in the same recording. For these reasons, we don't know the total number of speakers in the corpus and their gender/age.