littlelearner/LittleCurriculum-Chat

LittleCurriculum-Chat

Synthetic K–5 chat data generated with Gemini 2.5 Flash and filtered with the LittleCurriculum filter. Used to train the LittleLearner models.

Seeds are real documents rather than topic prompts, which keeps the topic distribution broad. The seed documents are not included here, so some questions assume context the reader does not have ("what is the main idea of this article?"). seed_id is the FineWeb-Edu document id, so the source text can be recovered by joining against LittleCurriculum.

Every row carries a messages column in [{role, content}] form alongside the structured fields.

from datasets import load_dataset
ds = load_dataset("littlelearner/LittleCurriculum-Chat", "general_knowledge")

Related data

For instruction fine-tuning we additionally used SmolTalk, MMLU auxiliary-train and GSM8K, each filtered with our pipeline. For GRPO we used grade-stratified synthetic problems from the MathCAMPS pipeline alongside GSM8K, again filtered with our pipeline. We do not redistribute these — they derive from public datasets and reproduce in one command:

python filter_k5.py --hf-dataset HuggingFaceTB/smol-smoltalk --out smoltalk_k5.parquet

In our runs this data was most useful mixed into pretraining and midtraining rather than reserved for a dedicated SFT stage.

License

ODC-By 1.0, inherited from FineWeb-Edu via LittleCurriculum. Generated with Google Gemini.

Citation

@misc{li2026littlelearner,
      title={LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure},
      author={Fanfei Li and Jana Zeller and Manuel Prada-Corral and Thaddäus Wiedemer and Prasanna Mayilvahanan and Ryan Cotterell and Wieland Brendel},
      year={2026},
      eprint={2608.13545},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2608.13545},
}