babylm-anon/stratified_10m_curriculum

Dataset Card for Stratified 10M Curriculum

This is a stratified split by domain of the raw datasets used by the 2024 BabyLM challange.

Sampled from the original datasets, equal amounts of tokens for each stage (C1,...C5).

Child-directed speech accounts for nearly half of the original dataset by word count. In preliminary experiments using a training data influence estimation method, this category was by far the most influential. This dataset enables us to investigate whether this is merely an artifact of this domain's proportion in the dataset, or if it is truly more useful for language modeling.

The order in which to load this dataset is provided as .pt files (e.g., curriculum.pt or random.pt).

Pretraining split (train)

Validation split (validation)

Dataset Sources

• —C1: Child Directed Speech
• —CHILDES (MacWhinney 2000)
• —C2: Children's Books
• —Children Stories Text Corpus (Bensaid et al. 2021)
• —Children's Book Test (Hill et al. 2016)
• —C3: Dialogue
• —OpenSubtitles (Lison and Tiedemann 2016)
• —Switchboard Dialog Act Corpus (Stolcke et al. 2000)
• —British NationalCorpus (BNC), dialogue portion
• —C4: Educational
• —Simple Wiki (Warstadt et al. 2023)
• —QED (Abdelali et al. 2014)
• —C5: Written English
• —Standardized Project Gutenberg Corpus (Gerlach and Font-Clos 2018)
• —Wikipedia (Warstadt et al. 2023)

Uses

To be used for pretraining experiments with a custom Trainer that samples in a predetermined order.