Datasetpublic

saidutta69/Odia-Web-Corpus-v5

Odia Web Corpus v5 The largest and cleanest Odia text corpus to date — 4.16 million deduplicated documents, 7.74 GB. Built by merging and thoroughly cleaning four source collections. Dataset Details Language: Odia (ISO 639-3: or) Format: 28 sharded Parquet files Total Size: 7.74 GB Total Documents: 4,162,804 License: CC-BY-SA-4.0 Cleaning Pipeline Stage Removed Description Deduplication 30.2% Exact MD5 hash match Short lines… See the full description on the dataset page: https://huggingface.co/datasets/saidutta69/Odia-Web-Corpus-v5.

sourceHugging Facecc-by-sa-4.0updated 12d agoView on Hugging Face

0likes298downloads

Dataset Card

Odia Web Corpus v5

<br>

The largest and cleanest Odia text corpus to date — 4.16 million deduplicated documents, 7.74 GB. Built by merging and thoroughly cleaning four source collections.

Dataset Details

—Language: Odia (ISO 639-3: or)
—Format: 28 sharded Parquet files
—Total Size: 7.74 GB
—Total Documents: 4,162,804
—License: CC-BY-SA-4.0

Cleaning Pipeline

Stage	Removed	Description
Deduplication	30.2%	Exact MD5 hash match
Short lines	6.2%	Fewer than 10 characters
Non-Odia	0.7%	No Odia Unicode characters
Low Odia ratio	0.01%	Less than 10% Odia chars
HTML + URLs	—	Stripped markup and links

Overall reduction: 37.2% (from 6.63M lines, 9.91 GB raw input)

Source Composition

Source	Description	Contribution
v4 corpus	Curated Odia web text	~3.1M lines
abhilash87	Odia Wikipedia + news scrape	~900K lines
culturax-odia	Common Crawl Odia subset	~500K lines
hemendra	Odia web text collection	~400K lines

Data Fields

Field	Type	Description
`text`	string	Cleaned, deduplicated Odia text

Usage

python

from datasets import load_dataset

ds = load_dataset("saidutta69/Odia-Web-Corpus-v5", split="train")
print(f"Total documents: {len(ds)}")
print(ds[0]["text"][:200])

Citation

bibtex

@misc{odia-web-corpus-v5,
  author = {Sai Dutta},
  title = {Odia Web Corpus v5},
  year = {2026},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/saidutta69/Odia-Web-Corpus-v5}}
}