CoolFace
Datasetpublic

nassimjp/zamai-pashto-clean-cpt

ZamAI Pashto Clean CPT This dataset is a hyper-cleaned, optimized, and fully deduplicated version of tasal9/ZamAI-Pashto-Mega-Dataset intended for Causal Language Modeling (CLM), pre-training, or fine-tuning text models in the Pashto language. 🛠️ Pipeline & Filtering Details Before processing the ~1.5 GB stream, an Internal Built-In Self-Test (BIST) was executed to verify environment I/O permissions, validate character ratio logic, and check connection… See the full description on the dataset page: https://huggingface.co/datasets/nassimjp/zamai-pashto-clean-cpt.

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face
0likes29downloads
Dataset Card

ZamAI Pashto Clean CPT

This dataset is a hyper-cleaned, optimized, and fully deduplicated version of tasal9/ZamAI-Pashto-Mega-Dataset intended for Causal Language Modeling (CLM), pre-training, or fine-tuning text models in the Pashto language.

🛠️ Pipeline & Filtering Details

Before processing the ~1.5 GB stream, an Internal Built-In Self-Test (BIST) was executed to verify environment I/O permissions, validate character ratio logic, and check connection constraints.

The processing pipeline applied the following steps:

  1. 1.URL & Email Stripping: Removed web artifacts, URLs (http/https/www), and electronic mail patterns.
  2. 2.Whitespace Normalization: Truncated consecutive tabs, newlines, and spaces into single tokens.
  3. 3.Character Ratio Validation: Calculated the exact density of native Pashto scripts—safeguarding unique characters like ښ and ځ. Items below a 0.2 ratio threshold were systematically filtered out.
  4. 4.MD5 Deduplication: Maintained a flat execution hash set to drop completely identical passages.

📊 Dataset Statistics

StatisticValue
Total Unique Pashto Lines Saved2,578,442
Duplicates Removed2,324,518 (~47.4% of raw text)
Skipped (Empty or Non-String)8
Skipped (Too Short, < 3 Chars)1
Skipped (Low Pashto Script Density)890
Average Text Length182.9 characters
Minimum Text Length7 characters
Maximum Text Length72,332 characters

🚀 Usage

You can load this optimized dataset seamlessly into your PyTorch/Hugging Face training pipelines:

python
from datasets import load_dataset

dataset = load_dataset("nassimjp/zamai-pashto-clean-cpt", split="train")

# Preview the first sample
print(dataset[0])