NbAiLab/NPSC_orto

Dataset Card for NbAiLab/NPSC

Table of Contents

• —Dataset Description
• —Dataset Summary
• —Data Fields
• —Dataset Creation
• —Statistics
• —Document Types
• —Languages
• —Publish Periode
• —Considerations for Using the Data
• —Social Impact of Dataset
• —Discussion of Biases
• —Other Known Limitations
• —Additional Information
• —Dataset Curators
• —Licensing Information
• —Citation Information

Dataset Description

• —Homepage: https://www.nb.no/sprakbanken/
• —Repository: https://www.nb.no/sprakbanken/ressurskatalog/oai-nb-no-sbr-58/
• —Paper: https://www.nb.no/sprakbanken/
• —Point of Contact: Per Erik Solberg

The Norwegian Parliamentary Speech Corpus (NPSC) is a speech corpus made by the Norwegian Language Bank at the National Library of Norway in 2019-2021. The NPSC consists of recordings of speech from Stortinget, the Norwegian parliament, and corresponding orthographic transcriptions to Norwegian Bokmål and Norwegian Nynorsk. All transcriptions are done manually by trained linguists or philologists, and the manual transcriptions are subsequently proofread to ensure consistency and accuracy. Entire days of Parliamentary meetings are transcribed in the dataset.

This repository contains a version of the NPSC in the 🤗 Dataset Format. Note that the official release of the dataset, which can be found in the repository of the Norwegian Language Bank, contains more information than the version found here, including word-level metadata, metadata about the speakers, and detailed documentation.

Extra Data

This version of the dataset contains extra data. For both the Norwegian Bokmål and the Norwegian Nynorsk part of the corpus it is added a machine generated text fields that attempts to be orthographic correct. In addition the Norwegian Bokmål text is translated to Nynorsk (and the other way around). The extra fields are named sentence_nob and sentence_nno. In addition any text in brackets, like <ee> is removed. In most cases this indicates filler words. Additionally "double words" are deleted. This info is not in the original dataset, and is mainly done to make the text closer to the text used in subtitles.

How to Use

# Loads the 16K Bokmål corpus in streaming mode
from datasets import load_dataset
data = load_dataset("NbAiLab/NPSC_orto", config="16K_mp3_bokmaal", streaming=True)

Dataset Summary

The NPSC dataset contains JSON lines with language training data. The data loader will add audio data to this structure. Here is an example json object:

{
"sentence_id": 49853,
"sentence_order": 0,
"speaker_id": 32,
"meeting_date": "20170110",
"speaker_name": "Olemic Thommessen",
"sentence_text": "Stortingets møte er lovlig satt",
"sentence_language_code": "nb-NO",
"text": "Stortingets møte er lovlig satt",
"sentence_nbo": "Stortingets møte er lovlig satt.",
"sentence_nno": "Stortingets møte er lovleg satt.",
"start_time": 320246, 
"end_time": 323590,
"normsentence_text": "Stortingets møte er lovlig satt",
"transsentence_text": "Stortingets møte er lovleg sett",
"translated": 1,
"audio": {"path": "audio/20170110-095504_320246_323590.wav","array": [.......]}
}

Data Fields

Initial Data Collection

The procedure for the dataset creation is described in detail in our paper.

Statistics

Considerations for Using the Data

This corpus contains speech data. All recordings are of members of Parliament in a public setting, and can be distributed without any restrains.

Dataset Creators and Curators

The content of the dataset was created by the Norwegian Language Bank (Språkbanken) at the National Library of Norway. Javier de la Rosa, Freddy Wetjen, Per Egil Kummervold, and Andre Kaasen all contributed in making this into a HuggingFace Dataset. Thanks to the HuggingFace team for assistance.

License

The sound and the transcriptions are released under the CC-ZERO-license. The curation of the HuggingFace Dataset is released under CC-BY-SA-3-license.

Citation Information

The following article gives detailed information about the corpus. Please refer to the article and this page if you are using this dataset:

@misc{solberg2022norwegian,
      title={The Norwegian Parliamentary Speech Corpus},
      author={Per Erik Solberg and Pablo Ortiz},
      year={2022},
      eprint={2201.10881},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}