Shaligram-Dewangan/FW_tokenized_20B

Reproducing GPT-2 for Indic Languages This Dataset is part of my project on "Reproducing GPT-2 for Indic Languages". Check out the main repository: https://github.com/Shaligram-Dewangan/GPT-2-for-Indic-Languages Dataset Description This is a tokenized text data for pre-training small sized language models and learning purposes. This dataset contains approximately 20 billion English + Hindi language tokens. It is tokenized using a custom BPE Tokeinzer. The English… See the full description on the dataset page: https://huggingface.co/datasets/Shaligram-Dewangan/FW_tokenized_20B.