Datasetpublic

zhzh98/babylm-opensub-bilingual-50M

babylm-opensub-bilingual-50M Bilingual OpenSubtitles training corpora (50M EN anchor, byte-premium partners). Interleaved sentence streams for aligned/offset mixing modes. Configs en_nld_aligned pair: en-nl / mode: aligned pairs selected: 7303811 anchor units: 55796159 mixed sentences: 14607622 en_nld_offset pair: en-nl / mode: offset pairs selected: 7303811 anchor units: 55796159 mixed sentences: 14570422… See the full description on the dataset page: https://huggingface.co/datasets/zhzh98/babylm-opensub-bilingual-50M.

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes77downloads

Dataset Card

babylm-opensub-bilingual-50M

Bilingual OpenSubtitles training corpora (50M EN anchor, byte-premium partners). Interleaved sentence streams for aligned/offset mixing modes.

Configs

`en_nld_aligned`

—pair: en-nl / mode: aligned
—pairs selected: 7303811
—anchor units: 55796159
—mixed sentences: 14607622

`en_nld_offset`

—pair: en-nl / mode: offset
—pairs selected: 7303811
—anchor units: 55796159
—mixed sentences: 14570422

`en_zho_aligned`

—pair: en-zh / mode: aligned
—pairs selected: 7534665
—anchor units: 52392020
—mixed sentences: 15069330

`en_zho_offset`

—pair: en-zh / mode: offset
—pairs selected: 7534665
—anchor units: 52392020
—mixed sentences: 15058126

`nld_zho_aligned`

—pair: nl-zh / mode: aligned
—pairs selected: 6557079
—anchor units: 36532627
—mixed sentences: 13114158

`nld_zho_offset`

—pair: nl-zh / mode: offset
—pairs selected: 6557079
—anchor units: 36532627
—mixed sentences: 13076958

Load

python

from datasets import load_dataset
ds = load_dataset("zhuojing-huang/babylm-opensub-bilingual-50M", "en_nld_aligned")