datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Subtitlesopen-subtitles-bitext-miningopen-subtitles-256s-bitext-miningyyets-subtitlesopen_subtitlesThis is a new collection of translated movie subtitles from http://www.opensubtitles.org/.
IMPORTANT: If you use the OpenSubtitle corpus: Please, add a link to http://www.opensubtitles.org/ to your website and to your reports and publications produced with the data!
This is a slightly cleaner version of the subtitle collection using improved sentence alignment and better language checking.
62 languages, 1,782 bitexts
total number of files: 3,735,070
total number of tokens: 22.10G
total number of sentence fragments: 3.35Ganime-subtitlesopen-subtitles-500-bitext-miningopen-subtitles-250-bitext-mininganime_subtitles_DialoGPTyyets-subtitles
人人影视字幕数据
source
人人影视二十年数据开源分享
人人影视成立之初就是想把自己翻译的字幕分享给更多爱好者，历时二十年人人影视翻译了几万集的影视字幕，本着字幕组分享的宗旨，我把全部字幕文件和对应数据库打包一起分享出来给大家。有需要的请下载保存，同时还有以前我们开发的字幕软件和源码也一起打包，还有我们没来得及开发的网站设计模板和HTML页面等等。评论区不知道让不让贴链接，我在几个主要自媒体平台的评论区都贴了链接，我们的用户名都叫：人人聊影视，大家自己找一下。 至于人人影视原来的程序其实技术太老太复杂，开源出来对普通人意义不大，这里就不提供了。现在开发APP和网站比以前更方便，有多端的统一框架，也有更好的语言环境，效率更高。
至于之前的人人影视数据库，是否可以从新开发转成海外影视这一领域的评分和查询网站，我还在研究可行性，大致也有了想法，过几天我把想法说一说，有兴趣的到时可以看一看。
（仅保留字幕文件）
百度网盘：http://t.cn/A6mSIYu6 提取码：rrys
（第三方）Resilio Sync… See the full description on the dataset page: https://huggingface.co/datasets/qundao/yyets-subtitles.open_subtitles_envi
OpenSubtitles
Source: https://huggingface.co/datasets/open_subtitles
Num examples: 3,505,276
Language: English
from datasets import load_dataset
load_dataset("tdtunlp/open_subtitles_envi")
Format for Translation task
def preprocess(
sample,
instruction_key="### Instruction:",
input_key="Input:",
response_key="<|endofprompt|>",
end_key="<|endoftext|>",
en2vi=True,
):
if en2vi:
if random.random() < 0.5:
instruction = "Translate the… See the full description on the dataset page: https://huggingface.co/datasets/vietgpt/open_subtitles_envi.JP_Holo_Subtitles_Seconds_Format_for_Ankitask1324_open_subtitles_te_en_translation
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1324_open_subtitles_te_en_translation
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1324_open_subtitles_te_en_translation.moroccan-darija-youtube-subtitles
Moroccan Darija YouTube Subtitles Dataset
This dataset contains subtitles from YouTube videos in Moroccan Darija, a colloquial Arabic dialect spoken in Morocco. The subtitles were collected from several popular Moroccan YouTube channels, providing a diverse set of transcriptions in the Darija language.
Dataset Description
The dataset is provided as a CSV file, where each row represents a YouTube video and contains the following columns:
video_id: The unique identifier of… See the full description on the dataset page: https://huggingface.co/datasets/bourbouh/moroccan-darija-youtube-subtitles.taiga_corpus_subtitles
Dataset Card for Taiga Corpus - TV Series Subtitles
This dataset contains subtitles extracted from various TV series. The original data is sourced from the Taiga Corpus. It consists of line-level subtitle information with precise timing and additional metadata including series title and episode information. The dataset is designed for tasks such as subtitle alignment, translation, and dialogue analysis.
Dataset Details
Each record in the dataset contains the following… See the full description on the dataset page: https://huggingface.co/datasets/Fascinat0r/taiga_corpus_subtitles.HowTo100M-subtitles-small
HowTo100M-subtitles-small
The subtitles from a subset of the HowTo100M dataset.
ERR-transcription-to-subtitlesThis dataset is created by Ilja Samoilov. In dataset is tv show subtitles from ERR and transcriptions of those shows created with TalTech ASR.
from datasets import load_dataset, load_metric
dataset = load_dataset('csv', data_files={'train': "train.tsv", \
"validation":"val.tsv", \
"test": "test.tsv"}, delimiter='\t')
scb_mt_enth_2020_and_open_subtitles
Dataset Card for "scb_mt_enth_2020_and_open_subtitles"
More Information needed
subtitles-en-no-similar-shorter
Open subtitles (EN-NO): similar and shorter parallels
361938 samples of subtitles/sentences where the Norwegian parallel is a maximum of 60% of the size of the English text.
Additionally, the sentences are filtered on similarity scores with a Norwegian SBERT model, with a minimum of 0.7 similarity (score based on observations from 10k samples).
The purpose of this dataset is to identify how generative models perform joint operations on translation and reduction of sentences.… See the full description on the dataset page: https://huggingface.co/datasets/tollefj/subtitles-en-no-similar-shorter.youtube_subtitlesgerman-opus-subtitles
German OPUS OpenSubtitles
Dataset Description
This dataset contains German movie and TV subtitles from the OPUS OpenSubtitles corpus. It provides a large collection of natural, conversational German text extracted from movie and TV show subtitles.
Key Features
141,565,623 lines of German dialogue
4.2 GB of clean text data
92.5% unique lines (low duplication rate)
Natural conversational German across diverse genres
Minimal contamination (0.2% English, 0.8% ALL… See the full description on the dataset page: https://huggingface.co/datasets/arnomatic/german-opus-subtitles.Subtitles-rag-answers-r1
Subtitles-rag-answers-r1
You should mask everything except the last turn. The only part that matters to teach the model is the last turn, as you are teaching it to always output thinking, no matter what the user feeds it.
It's setup to be trained like R1:
babylm-ar-subtitles
babylm-ara
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: ara
Script: Unknown
Number of Documents: 65951
Total Tokens: 399142332
Tokens Per Category
subtitles: 399142332 tokens
Data Fields
text: The document text
doc_id: Unique identifier for the document
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the data
script:… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-ar-subtitles.Subtitles-rag-questions-r1
Subtitles-rag-questions-r1
You should mask everything except the last turn. The only part that matters to teach the model is the last turn, as you are teaching it to always output thinking, no matter what the user feeds it.
It's setup to be trained like R1:
open_subtitles_en_nl
Dataset Card for OpenSubtitles
Dataset Summary
This dataset is a subset from the en-nl open_subtitles dataset.
It contains only subtitles of tv shows that have a rating of at least 8.0 with at least 1000 votes.
The subtitles are also ordered and appended into buffers several lengths, with a maximum of 370 tokens
as tokenized by the 'yhavinga/ul2-base-dutch' tokenizer.
Supported Tasks and Leaderboards
[More Information Needed]
Languages
The languages… See the full description on the dataset page: https://huggingface.co/datasets/yhavinga/open_subtitles_en_nl.Anime_subtitles_CN
Dataset Card for Dataset Name
This repo contains a csv file about anime subtitles crawl from open web.This dataset could be used for t2t,all the NLP projects expectionly of the anime domain.It's part.1,probably will have part.2.
Dataset Description
anime_subtitles.csv: Contains two features('name' and 'caption') and 4055 rows,about 400MB. Each name represent one season or movie, caption contaions all the dialogues that the characters speaks but no characters name or… See the full description on the dataset page: https://huggingface.co/datasets/cilyy/Anime_subtitles_CN.conversational-persian-subtitles
Conversational Persian Subtitles
Dataset name: Conversational Persian SubtitlesCollaboration: Maral Zarvani & Milad Ghashangi AgdamLicense: CC BY 4.0Hugging Face Repo: https://huggingface.co/datasets/Maral/conversational-persian-subtitles
1. Dataset Description
This dataset contains cleaned Persian subtitle lines from a wide variety of Korean TV series and films, each line reflecting informal, conversational dialogue. All markup (square brackets, timecodes,etc.) has… See the full description on the dataset page: https://huggingface.co/datasets/Maral/conversational-persian-subtitles.babylm-bg-subtitles
babylm-bg
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: bg
Script: Cyrillic
Category: subtitles
Source: Unknown
Age Estimate: n/a
Number of Documents: 43179
Total Tokens: 277270105
Tokens Per Category
subtitles: 277270105 tokens
Data Fields
text: The document text
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-bg-subtitles.babylm-pt-subtitles
babylm-pt
Dataset Description
This dataset is part of the BabyLM multilingual collection.
Dataset Summary
Language: pt
Script: Latin
Category: subtitles
Source: Unknown
Age Estimate: n/a
Number of Documents: 50869
Total Tokens: 356455068
Tokens Per Category
subtitles: 356455068 tokens
Data Fields
text: The document text
category: Type of content (e.g., child-directed-speech, educational, etc.)
data-source: Original source of the data… See the full description on the dataset page: https://huggingface.co/datasets/BabyLM-community/babylm-pt-subtitles.task1330_open_subtitles_en_te_translation
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1330_open_subtitles_en_te_translation
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1330_open_subtitles_en_te_translation.