arnizamani/Sindhi-texts-big-dataset

Sindhi Texts (big dataset)

A large plain-text corpus of Sindhi (سنڌي), assembled for pretraining language models. It combines material digitized by Sindhi literary institutions and forums, a Sindhi encyclopedia, newspaper archives, a classical dictionary, and the Sindhi portions of two web-crawl corpora.

3.19 GB, ~1.81 billion characters, ~390,000 documents across 9 sources. With a Sindhi-specific 12k SentencePiece tokenizer that is roughly 530M tokens (3.2–3.5 characters per token, measured per source; treat as an estimate).

Most of the corpus is one document per file. Two sources are exceptions, described below.

Contents

Character counts are exact for fineweb2, CC100-Sindhi, newspapers and quran.sindhsalamat.com; elsewhere they are extrapolated from seeded 40-file samples.

Two sources are not one-document-per-file

`fineweb2/fineweb2_snd.jsonl` — JSON Lines, one document per physical line, each object exactly {"text": "..."}. Only the text column of the source parquet shards is kept. JSONL rather than plain text because FineWeb2 documents contain blank lines, so a flattened .txt cannot be split back into documents: doing so yields 264,298 blocks for the real 260,662 documents. JSON escapes newlines inside the string, so here the line count is the document count.

`CC100-Sindhi/sd.txt` — kept in the upstream CC100 format: one document per blank-line-separated block, 81,062 in total. Because it arrives already flattened, its document boundaries carry the same ambiguity described above, so treat blank-line splitting as approximate.

Layout

sindhiadabiboard.org/{Litrature,mehran,Navel,Lateefyat,...}/BookN.txt
books.sindhsalamat.com/<id>.txt
sindhiana.org/<first letter>/<article title>.txt
quran.sindhsalamat.com/<translation>.txt
jamesindhilughaat/vol{1,2,3}/<section>/<file>.txt
newspapers/awami_awaz/{00..08}/aa_NNNNNN.txt
newspapers/sindhi_express/{00..02}/se_NNNNNN.txt
newspapers/kawish/<Author_Name>/kw_NNNNNN.txt
CC100-Sindhi/sd.txt
fineweb2/fineweb2_snd.jsonl
blogs/bhurgri.com.txt

sindhiana.org is sharded by the first letter of the article title, and the two newspaper wire services into numbered shards of 2,000 files, so that no directory exceeds the Hub's 10,000-file limit. kawish is a columnist archive and is grouped by author (75 authors). Newspaper files hold the headline on the first line, a blank line, then the article body.

All text files are UTF-8 with LF line endings and no BOM.

Loading

The .txt sources, one document per file:

from datasets import load_dataset

ds = load_dataset(
    "arnizamani/Sindhi-texts-big-dataset",
    data_files={"train": ["sindhiadabiboard.org/**/*.txt", "books.sindhsalamat.com/*.txt"]},
    sample_by="document",
)

FineWeb2, one document per line:

ds = load_dataset(
    "arnizamani/Sindhi-texts-big-dataset",
    data_files="fineweb2/fineweb2_snd.jsonl",
    split="train",
)

Or read it as a stream, without materializing 1.3 GB in memory:

import json
from huggingface_hub import hf_hub_download

path = hf_hub_download("arnizamani/Sindhi-texts-big-dataset",
                       "fineweb2/fineweb2_snd.jsonl", repo_type="dataset")
with open(path, encoding="utf-8") as fh:
    for line in fh:
        text = json.loads(line)["text"]

Provenance

Processing applied

• —Exact-duplicate removal. Byte-identical files were collapsed corpus-wide to one copy. Within the newspaper archives 1,858 duplicate articles were dropped before conversion — one outlet had scraped 1,655 URLs twice.
• —Dictionary conversion. The 91 dictionary files were extracted through the Word document piece table, so "fast save" leftovers and deleted text are excluded, as are header and footnote streams.
• —Newspaper conversion. One file per article; link, date and byline are deliberately kept out of the text. Articles under 200 characters and blank rows were dropped (238 rows). The columnist email address that one outlet appended to the end of most articles was removed (4,903 articles). Non-breaking spaces were normalised to plain spaces.
• —Filename hygiene. Sindhi quotation marks were stripped from sindhiana.org filenames; author directory names are ASCII with underscores.
• —1,022 empty files removed from sindhiana.org (failed scrapes).

Beyond that, no character filtering, Unicode normalisation or tokenizer-specific markup has been applied. Text is kept as extracted, including characters a downstream pipeline may prefer to strip.

Known limitations

• —Uneven quality. The two crawl-derived sources are 54% of all characters and are noisier than the curated book sources. Roughly 30% of CC100 documents are under 200 characters.
• —Newspapers are heavily under-represented. Sindhi has tens of daily newspapers, several of them publishing for decades, which puts the realistic body of published Sindhi newspaper text in the millions of articles. What is here — 26,534 articles from three outlets, under 3% of this corpus by characters — is well under 0.1% of that. Coverage is narrow in time too: every article carrying a date (2,919 of them) falls between 2014 and 2021, clustered in 2015–2018. Contemporary journalistic prose, regional reporting and current-affairs vocabulary are correspondingly thin, and newspapers are the most promising direction for growing this dataset.
• —Near-duplicates remain. Only byte-identical content was removed. Reprints across newspapers and boilerplate shared between crawled pages are still present.
• —The 14 Quran translations are 14 renderings of one source text, which over-weights that material unless downsampled. The set is also not final: two further translations are still to be downloaded from Sindh Salamat, and one tafsir (exegesis) work is pending and not yet available. Both will be added once available, so expect this source to grow — the tafsir in particular is a different genre, commentary rather than translation, and will not duplicate the existing text.
• —`sindhiana.org` is short-form and incomplete. Nearly 19,000 entries averaging under 1,600 characters, some of them one-line cross-references. The encyclopedia itself is also still a work in progress: roughly two volumes remain unpublished, so what is here is only the portion published so far and the coverage of later alphabet ranges is thinner than it will eventually be.
• —Dictionary gaps. Sections 44 and 48 are missing from volume 3, and three volume title pages are pure English. Both are upstream omissions.
• —Mixed script. Latin-script fragments (URLs, transliterations, English quotations) occur throughout, most heavily in the crawl-derived sources.
• —No train/validation/test split is provided; splitting is left to the consumer.

Version history

• —2.0 — July 2026. Added fineweb2 (260,662 documents), CC100-Sindhi (81,062 documents), newspapers (26,534 articles), jamesindhilughaat (91 dictionary sections) and blogs. Deduplicated corpus-wide, resharded sindhiana.org by first letter, removed 1,022 empty files. Grew from 1.28 GB to 3.19 GB.
• —1.0 — January 2026. sindhiadabiboard.org, sindhiana.org, quran.sindhsalamat.com, books.sindhsalamat.com.

Licensing and attribution

The license: mit field above covers the compilation, organisation and conversion work in this repository. It does not relicense the underlying texts, which come from several parties on differing terms:

• —fineweb2 derives from HuggingFaceFW/fineweb-2, released under the Open Data Commons Attribution License (ODC-By 1.0) and additionally subject to the Common Crawl Terms of Use. Attribution to FineWeb2 is required when using or redistributing it.
• —CC100-Sindhi derives from CC100, likewise built from Common Crawl and subject to its Terms of Use.
• —The book, encyclopedia, dictionary and newspaper material remains the copyright of its respective authors and publishers, and is included here to support research on a low-resource language. Rights holders who want material removed are welcome to open a discussion on this repository.

If you redistribute this dataset, carry these notices with it.

Citation

For this dataset:

@misc{abdul_rahim_nizamani_2026,
	author       = { Abdul Rahim Nizamani },
	title        = { Sindhi-texts-big-dataset (Revision 2106ece) },
	year         = 2026,
	url          = { https://huggingface.co/datasets/arnizamani/Sindhi-texts-big-dataset },
	doi          = { 10.57967/hf/9789 },
	publisher    = { Hugging Face },
	note         = { Version 2.0 }
}

Please also cite the upstream corpora this dataset incorporates. Their BibTeX entries are deliberately not reproduced here — copy them from the sources below:

• —FineWeb2 — Penedo et al., FineWeb2: One Pipeline to Scale Them All — Adapting Pre-Training Data Processing to Every Language, 2025, arXiv:2506.20920, <https://arxiv.org/abs/2506.20920>.
• —CC100 — Conneau et al., Unsupervised Cross-lingual Representation Learning at Scale, ACL 2020, pp. 8440–8451, <https://aclanthology.org/2020.acl-main.747>. CC100 also asks that CCNet be cited: Wenzek et al., CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data, LREC 2020, <https://aclanthology.org/2020.lrec-1.494/>.

The dictionary is a published work and is best cited directly: N. A. Baloch, A Comprehensive Dictionary of Classical & Current Sindhi (Jamé Sindhi Lughat), Sindhi Language Authority, first revised edition, 2004.