codelion/sutra-10B

Sutra 10B Pretraining Dataset

A high-quality pedagogical dataset designed for LLM pretraining, containing 10,193,029 educational entries totaling over 10 billion tokens. This is the largest dataset in the Sutra series, designed to demonstrate that dense, curated datasets can provide best-in-class pretraining performance for small language models.

Dataset Description

This dataset was generated using the Sutra framework, which creates structured educational content optimized for language model pretraining. Each entry is designed to maximize learning efficiency through:

• —Clear pedagogical structure: Content follows proven educational patterns
• —Cross-domain connections: Concepts are linked across disciplines
• —Varied complexity levels: From foundational (level 1) to advanced (level 10)
• —Quality-controlled generation: All entries meet minimum quality thresholds
• —Diverse content types: 33 different pedagogical formats
• —Rich metadata: Every entry annotated with 13 structured fields

Dataset Statistics

Domain Distribution

Content Type Distribution (Top 15)

Data Sources

Sutra-10B was created by scaling the same recipe used for Sutra-1B from 1 billion to 10 billion tokens. The core pedagogical content was generated using the Sutra framework, then mixed with several high-quality open datasets for diversity:

Data Fields

Each entry contains 13 structured fields:

Quality Assessment Sub-fields

Valid Domains (9)

mathematics, science, technology, language_arts, social_studies, arts_and_creativity, life_skills, philosophy_and_ethics, interdisciplinary

Valid Content Types (33)

concept_introduction, reasoning_demonstration, code_implementation, technical_documentation, tutorial, cross_domain_bridge, worked_examples, qa_pairs, common_misconceptions, meta_learning, synthesis, prerequisite_scaffolding, code_explanation, diagnostic_assessment, code_debugging, historical_context, research_summary, problem_set, case_study, analogy, experiment_design, proof, algorithm_analysis, data_analysis, ethical_analysis, comparative_analysis, creative_writing, debate_argument, practical_application, thought_experiment, visualization, system_design, review_summary

Data Cleaning

The dataset underwent comprehensive cleaning:

• —Deduplication: SHA-256 hash-based exact duplicate removal across all sources
• —Quality Filtering: Entries below quality_score 0.3 removed
• —Length Filtering: Entries shorter than 50 tokens or longer than 65,536 tokens removed
• —Garbage Detection: Repetitive content, control characters, non-English content filtered
• —Field Validation: All 13 fields validated and normalized

Metadata Generation

Metadata was generated using heuristic keyword-based classification:

• —Domain and content type classification via pattern matching and text analysis
• —Quality scores computed from text statistics (vocabulary diversity, structure, length)
• —Token counts computed using SmolLM2 tokenizer for accuracy

Usage

from datasets import load_dataset

# Load the full dataset
ds = load_dataset("codelion/sutra-10B", split="train")

# Stream for large-scale training
ds = load_dataset("codelion/sutra-10B", split="train", streaming=True)

# Filter by domain
math_ds = ds.filter(lambda x: x["domain"] == "mathematics")

# Filter by quality
high_quality = ds.filter(lambda x: x["quality_score"] > 0.7)

# Filter by complexity
beginner = ds.filter(lambda x: x["complexity_level"] <= 3)

Scaling Trajectory

Sutra-10B is the largest dataset in the Sutra series, scaling the original 1B recipe by 10x. When evaluated on SmolLM2-70M (69M parameters), benchmark performance remains consistent across scales, suggesting the model has reached its capacity ceiling. Larger models are expected to benefit more from the additional data and diversity.

Intended Use

This dataset is designed for:

• —LLM Pretraining: High-quality educational content for foundational model training
• —Domain-specific fine-tuning: Subset by domain for specialized training
• —Educational AI research: Studying pedagogical content generation
• —Curriculum learning: Progressive complexity for staged training
• —Small model optimization: Demonstrating data quality > quantity for small LMs

Related Datasets

• —sutra-1B: 1B token pretraining dataset
• —sutra-100M: 100M token subset
• —sutra-30k-seeds: Instruction prompts for post-training
• —sutra-magpie-sft: SFT dataset

Citation

@article{sharma2026sutra,
  title={Scaling Pedagogical Pretraining: From Optimal Mixing to 10 Billion Tokens},
  author={Sharma, Asankhaya},
  year={2026},
  url={https://huggingface.co/blog/codelion/scaling-pedagogical-pretraining-10-billion-tokens}
}

License

Apache 2.0