nis12ram/nisram-hindi-text-0.0

Dataset Card for nisram-hindi-text-0.0

Quick Summary

• —Language: Hindi
• —Size: ~618,000 raw text samples (206k from each source, before deduplication) and 602,000 after deduplication
• —Content: Short- to medium-length Hindi text (up to ~5000 characters), drawn from web-crawled corpora
• —Fields: One field text (string) containing the Hindi text
• —Sources:
• —mC4-Hindi-Cleaned-3.0
• —OSCAR-2301-Hindi-Cleaned-2.0
• —ai4bharat/sangraha (Hindi, verified)
• —Deduplication: Performed using a hashing-based script
• —Intended Use: Small-scale Hindi pretraining, or as input text for downstream tasks (e.g., NER, classification)

Dataset Description

The nisram-hindi-text-0.0 corpus is a curated collection of Hindi text samples, collected from three publicly available web-scraped datasets. Texts cover diverse topics and genres, excluding highly technical (STEM) content. Some samples may shift topics abruptly due to how they were scraped.

All text is in Devanagari script (Hindi) and is presented as a single field called text. No manual annotation or cleaning was done beyond deduplication.

Curation Rationale and Source Data

The dataset was created to support lightweight pretraining and downstream tasks in Hindi NLP. Each of the following three datasets contributed 206,000 examples:

• —[mC4-Hindi-Cleaned-3.0] A Hindi-cleaned version of the mC4 Common Crawl corpus. (License unspecified; CC-BY style likely.)

• —[OSCAR-2301-Hindi-Cleaned-2.0] A cleaned Hindi dataset from OSCAR. Licensed under Apache 2.0.

• —[AI4Bharat Sangraha (Hindi, verified)] A manually verified subset of AI4Bharat’s Sangraha corpus. Licensed under CC-BY-4.0.

After sampling, all entries were deduplicated using a hashing-based script to remove exact duplicates.

Dataset Structure

• —Split: Single train split
• —Format: JSONL or Parquet
• —Fields:
• —text (string): Hindi sentence or passage (max length ~5000 characters)

Example entry:

{
  "text": "विश्व पर्यावरण दिवस के अवसर पर कलेक्ट्रेट परिसर रोशनाबाद में पौधारोपण किया गया..."
}

Use Cases

• —Pretraining / Fine-tuning: Train transformer-based language models in Hindi
• —NER / Text Classification: Use as input text for Hindi downstream tasks
• —Unsupervised Tasks: Embedding learning, clustering, or generative modeling
• —Data Augmentation: Generate pseudo-labels or synthetic training samples

Known Limitations

• —⚠️ Topic shifts within samples — Some entries may contain multiple unrelated topics
• —⚠️ No STEM content — Scientific and technical language is underrepresented
• —⚠️ Web-scraped noise — Potential formatting errors, broken text, or informal language
• —⚠️ Bias and offensive content — No filtering for harmful language or personal data
• —⚠️ No manual review or anonymization — Use responsibly, especially in production

Licensing

This dataset is a mixture of samples from datasets under the following licenses:

• —Apache 2.0 — for OSCAR-2301-Hindi-Cleaned-2.0
• —CC-BY-4.0 — for AI4Bharat Sangraha (verified Hindi subset)
• —CC-BY-style (inferred) — for mC4-Hindi-Cleaned-3.0 (from Common Crawl)