CoolFace
Datasetpublic

Shaligram-Dewangan/FW_tokenized_20B

Reproducing GPT-2 for Indic Languages This Dataset is part of my project on "Reproducing GPT-2 for Indic Languages". Check out the main repository: https://github.com/Shaligram-Dewangan/GPT-2-for-Indic-Languages Dataset Description This is a tokenized text data for pre-training small sized language models and learning purposes. This dataset contains approximately 20 billion English + Hindi language tokens. It is tokenized using a custom BPE Tokeinzer. The English… See the full description on the dataset page: https://huggingface.co/datasets/Shaligram-Dewangan/FW_tokenized_20B.

sourceHugging Faceupdated 1y agoView on Hugging Face
1likes1.3kdownloads
Dataset Card

Reproducing GPT-2 for Indic Languages

This Dataset is part of my project on "Reproducing GPT-2 for Indic Languages". Check out the main repository: https://github.com/Shaligram-Dewangan/GPT-2-for-Indic-Languages

Dataset Description

This is a tokenized text data for pre-training small sized language models and learning purposes.

This dataset contains approximately 20 billion English + Hindi language tokens. It is tokenized using a custom BPE Tokeinzer. The English dataset is from Fineweb-Edu sample-10BT and Hindi dataset is from Fineweb-2 hin_Deva dataset.

Dataset Card

**Attribute****Details**
Total size~20.6 Billion tokens
Size on disk~40 GB
LanguagesEnglish and Hindi
English tokens~10.6 Billion
Hindi tokens~10 Billion
Mixing typeRandom mixing
TokenizerBPE tokenizer
Data SourceFineweb-v1 and Fineweb-v2

References

  1. 1.Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale, 2024. Blog · Fineweb-Edu Repo
  1. 1.Kydlicek et al. FineTasks: Finding signal in a haystack of 200+ multilingual tasks, 2024. Blog · Fineweb-2 Repo