datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
zhoblimphanzi-pinyinhanzi-structurevpswap-checkpoint-scores
VP-Swap checkpoint scores
Per-item correctness on the VP-Swap benchmark for nine models at twenty
points in training. This is the raw material behind Figures 4-6 of
Augustinian BabyLM (paper, code).
Layout
<model>/<revision>.jsonl, one line per benchmark item:
{"property": "color", "line": 6, "which": 1,
"pll_orig": -21.4213, "pll_swap": -23.9077, "correct": true}
property and line identify the source line in
eval/vpswap_bb24/vp_swap_<property>_pairs.txt; which… See the full description on the dataset page: https://huggingface.co/datasets/augustinian-babylm/vpswap-checkpoint-scores.babylm-2024-baby-cosmo-fine-100m@misc{charpentier2024gptbertboth,
title={GPT or BERT: why not both?},
author={Lucas Georges Gabriel Charpentier and David Samuel},
year={2024},
eprint={2410.24159},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.24159},
}
babylm-2024-baby-cosmo-fine-10m@misc{charpentier2024gptbertboth,
title={GPT or BERT: why not both?},
author={Lucas Georges Gabriel Charpentier and David Samuel},
year={2024},
eprint={2410.24159},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2410.24159},
}
babylm-curriculum-run07babylm-dino-embeddingsBabyLM-Pretokenisedzhoblimpbabylm_eng_distilled_1024This data is based on the babylm-eng dataset. It was processed and distilled through DeepSeek. Each entry was truncated to 3500 characters and then rewritten in simpler English.
The output tokens for each entry are limited to 1024 tokens.
This dataset was created with a strong focus on noise removal and simple language, making it suitable for tiny language models.
The following prompt details the instructions for the distillation process:
Instruction
You are an expert data curator… See the full description on the dataset page: https://huggingface.co/datasets/sixf0ur/babylm_eng_distilled_1024.babylm-curriculum-sliding-window-4bandsNLI-hiddenbabylm-curriculum-tiered-4bandsbabylm-curriculum-run05babylm-curriculum-run06babylm-curriculum-tiered-2bands