datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
open-subtitles-bitext-miningopen-subtitles-256s-bitext-miningyyets-subtitlesanime-subtitlesopen-subtitles-500-bitext-miningopen-subtitles-250-bitext-miningopen_subtitles_envi
OpenSubtitles
Source: https://huggingface.co/datasets/open_subtitles
Num examples: 3,505,276
Language: English
from datasets import load_dataset
load_dataset("tdtunlp/open_subtitles_envi")
Format for Translation task
def preprocess(
sample,
instruction_key="### Instruction:",
input_key="Input:",
response_key="<|endofprompt|>",
end_key="<|endoftext|>",
en2vi=True,
):
if en2vi:
if random.random() < 0.5:
instruction = "Translate the… See the full description on the dataset page: https://huggingface.co/datasets/vietgpt/open_subtitles_envi.task1324_open_subtitles_te_en_translation
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1324_open_subtitles_te_en_translation
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1324_open_subtitles_te_en_translation.moroccan-darija-youtube-subtitles
Moroccan Darija YouTube Subtitles Dataset
This dataset contains subtitles from YouTube videos in Moroccan Darija, a colloquial Arabic dialect spoken in Morocco. The subtitles were collected from several popular Moroccan YouTube channels, providing a diverse set of transcriptions in the Darija language.
Dataset Description
The dataset is provided as a CSV file, where each row represents a YouTube video and contains the following columns:
video_id: The unique identifier of… See the full description on the dataset page: https://huggingface.co/datasets/bourbouh/moroccan-darija-youtube-subtitles.taiga_corpus_subtitles
Dataset Card for Taiga Corpus - TV Series Subtitles
This dataset contains subtitles extracted from various TV series. The original data is sourced from the Taiga Corpus. It consists of line-level subtitle information with precise timing and additional metadata including series title and episode information. The dataset is designed for tasks such as subtitle alignment, translation, and dialogue analysis.
Dataset Details
Each record in the dataset contains the following… See the full description on the dataset page: https://huggingface.co/datasets/Fascinat0r/taiga_corpus_subtitles.HowTo100M-subtitles-small
HowTo100M-subtitles-small
The subtitles from a subset of the HowTo100M dataset.
ERR-transcription-to-subtitlesThis dataset is created by Ilja Samoilov. In dataset is tv show subtitles from ERR and transcriptions of those shows created with TalTech ASR.
from datasets import load_dataset, load_metric
dataset = load_dataset('csv', data_files={'train': "train.tsv", \
"validation":"val.tsv", \
"test": "test.tsv"}, delimiter='\t')
scb_mt_enth_2020_and_open_subtitles
Dataset Card for "scb_mt_enth_2020_and_open_subtitles"
More Information needed
subtitles-en-no-similar-shorter
Open subtitles (EN-NO): similar and shorter parallels
361938 samples of subtitles/sentences where the Norwegian parallel is a maximum of 60% of the size of the English text.
Additionally, the sentences are filtered on similarity scores with a Norwegian SBERT model, with a minimum of 0.7 similarity (score based on observations from 10k samples).
The purpose of this dataset is to identify how generative models perform joint operations on translation and reduction of sentences.… See the full description on the dataset page: https://huggingface.co/datasets/tollefj/subtitles-en-no-similar-shorter.youtube_subtitlesgerman-opus-subtitles
German OPUS OpenSubtitles
Dataset Description
This dataset contains German movie and TV subtitles from the OPUS OpenSubtitles corpus. It provides a large collection of natural, conversational German text extracted from movie and TV show subtitles.
Key Features
141,565,623 lines of German dialogue
4.2 GB of clean text data
92.5% unique lines (low duplication rate)
Natural conversational German across diverse genres
Minimal contamination (0.2% English, 0.8% ALL… See the full description on the dataset page: https://huggingface.co/datasets/arnomatic/german-opus-subtitles.Subtitles-rag-answers-r1
Subtitles-rag-answers-r1
You should mask everything except the last turn. The only part that matters to teach the model is the last turn, as you are teaching it to always output thinking, no matter what the user feeds it.
It's setup to be trained like R1:
babylm-ar-subtitles
babylm-ara
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: ara
Script: Unknown
Number of Documents: 65951
Total Tokens: 399142332
Tokens Per Category
subtitles: 399142332 tokens
Data Fields
text: The document text
doc_id: Unique identifier for the document
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the data
script:… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-ar-subtitles.Subtitles-rag-questions-r1
Subtitles-rag-questions-r1
You should mask everything except the last turn. The only part that matters to teach the model is the last turn, as you are teaching it to always output thinking, no matter what the user feeds it.
It's setup to be trained like R1:
open_subtitles_en_nl
Dataset Card for OpenSubtitles
Dataset Summary
This dataset is a subset from the en-nl open_subtitles dataset.
It contains only subtitles of tv shows that have a rating of at least 8.0 with at least 1000 votes.
The subtitles are also ordered and appended into buffers several lengths, with a maximum of 370 tokens
as tokenized by the 'yhavinga/ul2-base-dutch' tokenizer.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
The languages… See the full description on the dataset page: https://huggingface.co/datasets/yhavinga/open_subtitles_en_nl.Anime_subtitles_CN
Dataset Card for Dataset Name
This repo contains a csv file about anime subtitles crawl from open web.This dataset could be used for t2t,all the NLP projects expectionly of the anime domain.It's part.1,probably will have part.2.
Dataset Description
anime_subtitles.csv: Contains two features('name' and 'caption') and 4055 rows,about 400MB. Each name represent one season or movie, caption contaions all the dialogues that the characters speaks but no characters name or… See the full description on the dataset page: https://huggingface.co/datasets/cilyy/Anime_subtitles_CN.conversational-persian-subtitles
Conversational Persian Subtitles
Dataset name: Conversational Persian SubtitlesCollaboration: Maral Zarvani & Milad Ghashangi AgdamLicense: CC BY 4.0Hugging Face Repo: https://huggingface.co/datasets/Maral/conversational-persian-subtitles
1. Dataset Description
This dataset contains cleaned Persian subtitle lines from a wide variety of Korean TV series and films, each line reflecting informal, conversational dialogue. All markup (square brackets, timecodes,etc.) has… See the full description on the dataset page: https://huggingface.co/datasets/Maral/conversational-persian-subtitles.babylm-bg-subtitles
babylm-bg
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: bg
Script: Cyrillic
Category: subtitles
Source: Unknown
Age Estimate: n/a
Number of Documents: 43179
Total Tokens: 277270105
Tokens Per Category
subtitles: 277270105 tokens
Data Fields
text: The document text
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-bg-subtitles.babylm-pt-subtitles
babylm-pt
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: pt
Script: Latin
Category: subtitles
Source: Unknown
Age Estimate: n/a
Number of Documents: 50869
Total Tokens: 356455068
Tokens Per Category
subtitles: 356455068 tokens
Data Fields
text: The document text
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the data… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-pt-subtitles.task1330_open_subtitles_en_te_translation
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1330_open_subtitles_en_te_translation
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1330_open_subtitles_en_te_translation.babylm-fa-subtitles
babylm-fa
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: fa
Script: Latin
Category: subtitles
Source: Unknown
Age Estimate: n/a
Number of Documents: 41312
Total Tokens: 249245044
Tokens Per Category
subtitles: 249245044 tokens
Data Fields
text: The document text
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the data… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-fa-subtitles.babylm-de-subtitles
babylm-de
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: de
Script: Latin
Category: subtitles
Source: Unknown
Age Estimate: n/a
Number of Documents: 32073
Total Tokens: 224733295
Tokens Per Category
subtitles: 224733295 tokens
Data Fields
text: The document text
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the data… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-de-subtitles.task1329_open_subtitles_en_hi_translation
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1329_open_subtitles_en_hi_translation
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1329_open_subtitles_en_hi_translation.Subtitles-rag-answers-qwq-all-kcppbabylm-10M-open-subtitles
Dataset Card for "babylm-10M-open-subtitles"
More Information needed