daichi812/babyloop-mm-corpus

Datasheet: babyloop Multimodal Training Corpus (BabyLM 2026, Strict track)

Datasheet format follows Gebru et al. (2021), "Datasheets for Datasets," abridged to the sections relevant for a training-corpus release. All counts below are measured, not estimated; provenance manifests are versioned in this repository under [`manifests/`](./tree/main/manifests).

Summary

Motivation

This corpus was assembled to test whether looped transformer architectures (K=4) mitigate the BLiMP degradation induced by multimodal (vision+language) training under the BabyLM Strict 100M-word budget. BabyLM 2026 removed the dedicated Multimodal track and allows paired image–text data inside Strict; the image–text data referenced by the 2026 call is the OSF-hosted multimodal release from the earlier multimodal track — confirmed by the organizers (#ask-organizers, 2026-07-04/06), who noted participants are free to use it; no 2026-specific multimodal dataset exists on the BabyLM Hugging Face organization as of 2026-07-03. We combine the official 2026 text corpus with that multimodal caption/feature release, split 50/50 by word count so that text and grounded captions each receive half the budget.

Composition

Text portion (50M words) — text/train_50M.txt

• —Source: `BabyLM-community/BabyLM-2026-Strict` (Hugging Face), train split, revision 9e57baa, downloaded 2026-06-17. The full official corpus contains 100,000,000 whitespace words over 11,601,896 non-empty lines, released as six domain-blocked files.
• —Selection: the release is concatenated in alphabetical file order (verified line-by-line over all 11,601,896 lines with zero mismatches); our pipeline reads this concatenation sequentially and stops when the next line would exceed the 50M-word text budget. This yields a deterministic prefix of 49,999,988 words / 6,785,399 lines, hosted here as text/train_50M.txt.
• —Domain composition of the prefix (measured) — because the release is domain-blocked rather than shuffled, the prefix does not preserve the official domain mixture:

Full measurement (per-domain line/word counts, boundary location, verification method, HF revision) is recorded in `manifests/text_domain_boundaries.json`.

• —Preprocessing for training: tokenized with a frozen BPE tokenizer (vocab 16,000; trained once on the full 100M-word text-only corpus and reused across all conditions for comparability), packed into 512-token blocks (96,742,400 tokens; 188,950 blocks).

Caption portion (50M words, image-grounded) — captions/captions_50M.jsonl

• —Source: official BabyLM multimodal release, OSF project **`ad7qg`**, directory multimodal_data/ (downloaded 2026-06-21):
• —local_narr_captions.json — Localized Narratives captions, 767,736 pairs
• —cc_3M_captions.json — Conceptual Captions (CC3M) captions, 2,306,312 pairs
• —total available: 3,074,048 caption–feature pairs
• —Selection: captions are read in release order (all LN first, then CC3M, preserving row order within each file); reading stops when the next caption would exceed the 50M-word caption budget. This consumes 3,037,190 captions / 49,999,996 words: all 767,736 LN captions plus the first 2,269,454 CC3M captions (verified by direct count; the trailing 36,858 CC3M captions are unused).
• —Format: one JSON record per line, {"image_id": "ln_<row>" | "cc_<row>", "caption": "..."}. The numeric suffix of image_id is the row index within the corresponding source file of the OSF release, which is also the row index into the corresponding DINOv2 feature array (see Visual features).
• —Integrity: 0 captions skipped for missing features; 0 captions truncated (captions are stored per image–caption record, not packed, so that the visual prefix binds to its caption; a tail-truncation policy exists for captions exceeding 511 tokens but was never triggered).

Visual features

• —Precomputed frozen DINOv2 ViT-B/14 global CLS vectors (one 768-dim vector per image, V=1), distributed in the official OSF release (local_narr_dino_v2_states.npy, cc_3M_dino_v2_states_{1,2}of2.npy). They are not re-hosted here; download them from https://osf.io/ad7qg.
• —Caption ↔ feature correspondence is by row index within each source file (encoded in image_id).
• —Raw images are not part of this corpus and were never downloaded or used; only the official precomputed features enter training.

Word counting

Whitespace-split tokens, per the official BabyLM budget rule, applied identically to text lines and captions. Caption words count toward the 100M budget. Budget compliance is enforced programmatically (WordBudgetTracker, covered by unit tests) — total: 49,999,988 + 49,999,996 = 99,999,984 ≤ 100,000,000.

Collection & preprocessing process

The corpus is a deterministic function of the two official releases and the configuration word_budget=100,000,000, text_caption_ratio=0.5. No random sampling, filtering heuristics, or content modification (beyond whitespace stripping of line ends) is applied.

The assembled files hosted here were produced by the following rule and verified against the manifests:

1. 1.Text: concatenate the six official files in alphabetical order (bnc_spoken, childes, gutenberg, open_subtitles, simple_wiki, switchboard); read line by line (strip; skip empty); stop when the next line would exceed 50,000,000 words (the boundary line is excluded).
2. 2.Captions: read local_narr_captions.json then cc_3M_captions.json in row order; stop when the next caption would exceed 50,000,000 words.

Manifests recording source, download dates, revisions, and exact counts are mirrored in this repository under `manifests/`: text_SOURCE.json, text_domain_boundaries.json, text_tokenized_meta.json, captions_SOURCE.json, mm_text_meta.json, mm_captions_meta.json. The assembly code repository (download / preprocessing / verification scripts): https://github.com/daichi8120/babyloop.

Distribution / download

• —This repository hosts the assembled corpus: text/train_50M.txt (text portion) and captions/captions_50M.jsonl (caption portion), plus provenance manifests.
• —Text portion source: https://huggingface.co/datasets/BabyLM-community/BabyLM-2026-Strict (official 2026 release; revision 9e57baa used here)
• —Caption + feature source: https://osf.io/ad7qg (official BabyLM multimodal release, multimodal_data/); the DINOv2 feature files required for multimodal training remain at this official OSF source.
• —Both components are official BabyLM releases; this document specifies the exact mixture, and the deterministic recipe above reproduces the hosted files bit-exactly from the two official sources. Per organizer guidance (#ask-organizers, 2026-07-04: links plus a deterministic recipe are acceptable; sharing on HF is preferred), the assembled text and caption files are hosted here directly.

Licensing & redistribution

Both components inherit the terms of their official BabyLM releases. Localized Narratives annotations are CC BY 4.0; Conceptual Captions text is distributed under Google's Conceptual Captions terms. Raw images (COCO / Open Images / CC3M) are not redistributable and are not part of this corpus; following the BabyLM multimodal release design, only captions and precomputed DINOv2 features are used. The 2026 text corpus was detoxified by the organizers (Trhlik et al., 2026). The caption data was used exactly as distributed in the OSF project as of 2026-06-21; the organizers confirmed (#ask-organizers, 2026-07-04/06) that the 2026 detoxification procedure applied to the text-only corpora and that the multimodal data is unchanged.

Uses

Training corpus for the multimodal cells of a 2×2 (architecture × modality) study: `babyloop-std-mm` and `babyloop-loop-mm` (BabyLM 2026 leaderboard, Strict track), and the analysis-only variants std_mm_naive (identity connector) and loop_mm_prefix (injection-matched clean baseline). The text-only cells (babyloop-std-text, babyloop-loop-text) use the unmodified official 2026 Strict corpus (full 100M words) and require no datasheet.

Note for users of this corpus: the text half's domain composition differs from the official mixture (see Composition); comparisons against models trained on the full official corpus conflate this register shift with any multimodal effects.

Loading

Both configs are served by the text builder (a single repository cannot mix builder types), so caption records arrive as raw JSON lines — parse them with json.loads:

import json
from datasets import load_dataset

text = load_dataset("daichi812/babyloop-mm-corpus", "text", split="train")
# {"text": "<one corpus line>"}

caps = load_dataset("daichi812/babyloop-mm-corpus", "captions", split="train")
caps = caps.map(lambda r: json.loads(r["text"]), remove_columns=["text"])
# {"image_id": "ln_0", "caption": "..."}

Maintenance & contact

Maintained by daichi812 (Hugging Face). Questions and issues via this repository's Discussions. Code repository: https://github.com/daichi8120/babyloop. The associated paper will be linked here after the double-blind review period.