frikishaan/PrimeCorpus-1B

PrimeCorpus-1B

PrimeCorpus-1B is a curated 1-billion tokens text dataset created for training small and mid-scale language models. It focuses on educational, encyclopedic, and narrative domains to provide a balanced learning signal, and is created specifically for learning and experimentation.

Composition

**Note** - Token counts are measured using a GPT-2 tokenizer.

Processing

• —Markdown syntax removed (headers, bold, italic, etc.).
• —Gutenberg texts aggressively cleaned due to heavy noise and structural artifacts.
• —Any non-english data is removed
• —Included only fineweb-edu samples with a score greater than 4.

Intended Use

• —Training and evaluation of small language models.
• —Designed for pre-training GPT-2–style models focused on prose generation, not conversational training.
• —Experiments in architecture variations, scaling laws, and curriculum strategies.
• —Educational and research-oriented projects.

Limitations

• —Not designed for production-grade or safety-critical applications.
• —Domain coverage is intentionally narrow.
• —Source biases remain.

License

Each component retains its original license. Users must ensure compliance with the respective source licenses when redistributing or training models.