hasankursun/bulgarian-corpus-33b

Bulgarian Corpus 33B (BC-33B)

Dataset Summary

The Bulgarian Corpus 33B (BC-33B) is a massive-scale, deduplicated, and cleaned dataset designed for training Foundation Models in Bulgarian. Comprising approximately 33.4 Billion tokens (measured with Qwen 2.5/Llama-3 tokenizer), it represents one of the largest open-source resources for Bulgarian LLM pretraining.

The dataset is engineered for a modern two-stage training pipeline:

1. 1.Pretrain Subset (~29.3B Tokens): A diverse mix of high-quality web data, encyclopedic knowledge, and scientific abstracts.
2. 2.SFT Subset (~4.1B Tokens): A curated collection of instruction-following, chat, and multitask data, strictly filtered to remove alignment artifacts.

Training Recommendation: With ~33B unique high-quality tokens, we recommend training for 3 Epochs over the pretrain subset to achieve optimal convergence for models in the 7B-8B parameter range (effectively ~90B training tokens).

Dataset Statistics

Estimates based on Qwen 2.5 / Llama-3 Tokenization.

Data Structure

1. Pretraining Subset (pretrain)

Optimized for high-throughput streaming with libraries like datatrove, nanotron, or torchtune.

2. SFT Subset (sft)

Optimized for "Instruction Pretraining" or Fine-Tuning (Axolotl/LLaMA-Factory compatible).

Data Composition

This corpus was built using a Quality-First strategy, blending massive web scale with high-density scientific and encyclopedic data.

Processing Pipeline

This dataset was engineered for Foundation Model training standards:

1. 1.Normalization: Multiple raw data sources were mapped to a single unified schema.
2. 2.PII Sanitization:
3. 3.Regex Cleaning: Automated removal of Email addresses, IPv4 addresses, and Bulgarian phone numbers (e.g., +359..., 088...).
4. 4.DB-Assisted Deduplication:
5. 5.Exact deduplication (MD5 hashing) was performed across the entire collection.
6. 6.Priority Strategy: High-quality sources (Wiki/Science) were processed first to claim ownership of duplicate text, ensuring the highest quality version is kept.
7. 7.Quality Filtering (SFT):
8. 8.The SFT subset was scrubbed of "poison" rows (e.g., where the assistant replies "None", "null", or refuses to answer due to alignment errors).
9. 9.Sharding: Data is split into ~200k row Parquet shards for optimal download and streaming speeds.

Limitations

• —Web Bias: A significant portion of the data (FineWeb) comes from the open internet and may reflect societal biases found in Bulgarian web content.
• —Translation Artifacts: Some SFT data is machine-translated or aligned; while we filtered obvious errors, some translation artifacts may remain.

Citation & Attribution

If you use this dataset in your research or product, please cite:

@misc{bulgariancorpus33b,
  author = {Hasan KURŞUN},
  title = {Bulgarian Corpus 33B},
  year = {2025},
  url = {[https://huggingface.co/datasets/hasankursun/bulgarian-corpus-33b](https://huggingface.co/datasets/hasankursun/bulgarian-corpus-33b)}
}