nassimjp/pashto-warmup-tokens

Pashto Warmup Tokens Dataset

This dataset contains a curated, deduplicated collection of high-quality, contextually accurate Pashto linguistic examples. It maps structural language tasks directly to the most critical vocabulary tokens in Pashto, providing a reliable corpus for token warmup, instruction tuning, evaluation, and post-OCR text correction workflows.

Dataset Summary

The initial release consists of 4,087 verified entries targeting high-frequency and structurally complex Pashto tokens (derived from a systematic vocabulary expansion tracking up to 20K tokens). Each row is structured as a standard training tuple designed to reinforce correct orthography, morphological variants, and syntactic application in Pashto NLP pipelines.

• —Repository Owner: nassimjp
• —Current Size: 4,087 rows (Expanding to 20K)
• —Format: JSON Lines (.jsonl)
• —Primary Focus: Pashto grammar, token stabilization, meaning-generation, and lexical definitions.

Dataset Structure

The dataset follows a standardized jsonl configuration with the following schema:

{
  "system": "ته د پښتو ژبې مرستندوی یې. په پښتو ځواب ورکړه.",
  "instruction": "د 'کلمه' کلمه په یوه معنی لرونکې پښتو جمله کې وکاروه.",
  "output": "دلته په پښتو ژبه کې یوه نمونه جمله ده."
}

Fields:

• —`system`: System prompt enforcing the agent's role as a Pashto language assistant.
• —`instruction`: The targeted linguistic prompt (e.g., requesting a definition or a contextual sentence for a specific token).
• —`output`: The generated, validated, and deduplicated Pashto response demonstrating accurate native morphology and orthography.

Intended Use Cases

1. 1.Token Warmup & Embedding Alignment: Stabilizing raw token distributions in LLMs for low-resource languages.
2. 2.Instruction Fine-Tuning: Teaching generative models how to define, parse, and utilize native Pashto terms in context.
3. 3.OCR Post-Processing: Providing linguistic priors to correct misread characters or out-of-vocabulary anomalies from OCR engines.

Roadmap & Expansion

• —[x] Phase 1: Core pipeline verification and deduplication (4,087 records).
• —[ ] Phase 2: Scaling dataset features to match the full pashto_top_20k_tokens.txt corpus.
• —[ ] Phase 3: Integrating dialectal variations and classical morphological edge cases.

License

This dataset is distributed under the MIT License. You are free to share, modify, and build upon this data for both commercial and non-commercial machine learning research.