Shaligram-Dewangan/FW_tokenized_20B
Reproducing GPT-2 for Indic Languages This Dataset is part of my project on "Reproducing GPT-2 for Indic Languages". Check out the main repository: https://github.com/Shaligram-Dewangan/GPT-2-for-Indic-Languages Dataset Description This is a tokenized text data for pre-training small sized language models and learning purposes. This dataset contains approximately 20 billion English + Hindi language tokens. It is tokenized using a custom BPE Tokeinzer. The English… See the full description on the dataset page: https://huggingface.co/datasets/Shaligram-Dewangan/FW_tokenized_20B.
Conversations for this repository live on Hugging Face.
CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.
Open discussions on Hugging Face