arnizamani/Sindhi-texts-big-dataset
Sindhi Texts (big dataset) A large plain-text corpus of Sindhi (سنڌي), assembled for pretraining language models. It combines material digitized by Sindhi literary institutions and forums, a Sindhi encyclopedia, newspaper archives, a classical dictionary, and the Sindhi portions of two web-crawl corpora. 3.19 GB, ~1.81 billion characters, ~390,000 documents across 9 sources. With a Sindhi-specific 12k SentencePiece tokenizer that is roughly 530M tokens (3.2–3.5 characters per… See the full description on the dataset page: https://huggingface.co/datasets/arnizamani/Sindhi-texts-big-dataset.
Remove upstream BibTeX blocks from citation section
Sync corpus (1/1): README.md
Sync corpus (2/2): .gitattributes
Sync corpus (1/2): CC100-Sindhi
Sync corpus (1/1): fineweb2
Sync corpus (12/12): newspapers
Sync corpus (11/12): newspapers
Sync corpus (10/12): newspapers
Sync corpus (9/12): newspapers
Sync corpus (8/12): newspapers
Sync corpus (7/12): newspapers
Sync corpus (6/12): newspapers
Sync corpus (5/12): newspapers
Sync corpus (4/12): newspapers
Sync corpus (3/12): newspapers
Sync corpus (2/12): newspapers
Sync corpus (1/12): newspapers
Sync corpus (2/14): newspapers
Sync corpus (1/14): newspapers
Add 4 sources, dedup corpus, reshard sindhiana.org by letter
Upload 93 files
Upload 28 files
Upload 173 files
Update README.md
Upload 212 files
Upload 85 files
Upload 236 files
Upload 60 files
Upload 21 files
Upload 2 files
Upload 14 files
Update README.md
initial commit