dzur658/opus-es-monologues

OPUS Spanish Monologues

A dataset that captures monologues from the Spanish Open Subtitles Project dump and undergoes light cleaning. Monologues retained in this dataset are intances in the raw .txt dump where a single speaker is uninterrupted for more than 100 words. The dataset consists of monologues from the 2013, 2016, and 2018 OPUS Spanish monolingual datasets.

Quick Dataset Facts

• —Contains 1,481 documents
• —Each document averages ~241.2 words
• —The dataset contains a total of 357,244 words

Intended Use Case

This dataset is intended to be used as part of a Continued Pretraining corpus for exposing a base LLM to informal and conversational Spanish.

Dataset Map

The dataset is a parquet file consisting of columns with the main content for Continued Pretraining and associated metadata.

• —content: The cleaned monologue text data.
• —document_ref: A concatenated string stating where the document emerged from, for example: opus-es-2016-line-16 tells us the document originated from the 2016 OPUS dump, and began on line 16.
• —metadata: A dictionary with three keys: family, source_file, and year. family is the general OPUS family (since all the data was sourced from OPUS ES monolingual dumps this remains the same throughout this dataset). source_file states which OPUS file the document originated from. Finally, year links the document to the associated OPUS dataset for that year.
• —line_id: The line number in the original OPUS text file the monologue begins on.
• —file_name: The name of the file the document originated from.
• —word_count: How many Spanish words exist in the document.
• —language: An artificat appended by the language identification step by the fasttext model. The first value is the confidence of the model, and the second value is the associated language code (ES).
• —_curator_dedup_id & __index_level_0__: Artifiacts appended by the NeMo Curator Deduplication process.

Dataset Creation

Download & Extract

First all 3 datasets are downloaded from the Open Subtitles Project dump. Then, the files are split on speaker turns which are indicated by a - in the raw .txt file.

Cleaning

The resulting data is then lightly cleaned. Regex filters are implemented to remove square brackets ([]), timestamps, embedded timestamps, html tags and elements. Whitespace normalizaton is also applied. Next, monologues are filtered for by only retaining documents that consist of 100 Spanish words or more. Finally, each document passes through a language identification filter, with each document needing to obtain a minimum langid score of 0.7 from the fasttext model.

Deduplication

Fuzzy deduplication (MinHash/LSH) is used to identify and remove duplicate documents. Due to combining multiple OPUS drops of the same language from different years we do this to ensure each document is truly unique, providing an equally weighted dataset for all documents.

GitHub

This dataset is the result of a NeMo Curator pipeline. The full code of the pipeline that reproduces this dataset can be found here.

Acknowledgements

Raw data is provided by the OpenSubtitles Project, thank you!