BaoMinhTran/mini-fineweb-edu

Mini FineWeb-Edu (CC-MAIN-2025-26)

Mini FineWeb-Edu (CC-MAIN-2025-26) is a lightweight subset of the original FineWeb-Edu dataset. To support different development and research workflows, the dataset is released in five progressively larger subsets: debug, tiny, small, base, and full.

The dataset is designed to provide a compact and practical corpus for developing and evaluating small language models (SLMs) and language model training pipelines, while significantly reducing storage and preprocessing requirements compared to the full FineWeb-Edu dataset.

Available Subsets

This dataset is organized into multiple subsets to support different stages of language model development and experimentation.

Each subset preserves the original document ordering from the CC-MAIN-2025-26 snapshot. Smaller subsets are strict prefixes of larger subsets (e.g., tiny contains all samples in debug, small contains all samples in tiny, etc.), allowing experiments to scale naturally without changing the underlying data distribution.

Intended Uses

This dataset is suitable for:

• —Pretraining Small Language Models (SLMs)
• —Verifying and debugging LLM/SLM training pipelines
• —Rapid experimentation with tokenizer training and vocabulary analysis
• —Benchmarking data loading, preprocessing, and distributed training systems
• —Educational purposes for learning large-scale language model pretraining
• —Reproducible research on data curation and corpus processing
• —Running experiments on consumer hardware or cloud platforms with limited storage and compute resources

Data Source

This dataset is derived from the CC-MAIN-2025-26 subset of the original FineWeb-Edu dataset.

Original dataset:

https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

Modifications

Compared to the original FineWeb-Edu dataset, this release:

• —contains only a small subset of the CC-MAIN-2025-26 snapshot;
• —preserves the original document ordering;
• —does not perform random sampling;
• —does not modify document contents.

License

This dataset is a derived work of FineWeb-Edu and is distributed under the Open Data Commons Attribution License (ODC-By) v1.0, following the licensing terms of the original dataset.

Please refer to the original dataset for additional licensing information.

Citation

This dataset is a derived subset of FineWeb-Edu. If you use this dataset in your work, please cite the original FineWeb-Edu dataset:

@misc{lozhkov2024fineweb-edu,
    author       = { Lozhkov, Anton and Ben Allal, Loubna and von Werra, Leandro and Wolf, Thomas },  
    title        = { FineWeb-Edu: the Finest Collection of Educational Content }, 
    year         = 2024,  
    url          = { https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu },  
    doi          = { 10.57967/hf/2497 },
    publisher    = { Hugging Face }
}