harims95/hobbylm-longctx-2p5b

HobbyLM-1B Long-Context Extension Corpus

2.5B GPT-2 tokens staged for extending harims95/hobbylm-1b-hf's context window from 1024 tokens to 4096, then 8192.

Format

Each <source>_NNNNNN.bin file is the same flat-binary format used throughout HobbyLM training: a 256 x int32 header (magic=20240520, version=1, num_tokens), followed by the token stream as uint16 GPT-2 token ids.

The flat .bin files do not encode document boundaries on their own. Each source also ships a <source>_index.json sidecar (compact, one entry per document: shard file, offset, length) and a <source>_docs.jsonl (the same data plus per-document metadata). Use these with hobbylm.data.boundary_aware_data_generator, which packs multiple documents into each training row and builds a block-diagonal attention mask + resets RoPE position ids at every document boundary (hobbylm.model.build_block_diagonal_bias / positions_from_segments) so no document ever attends into another's tokens, regardless of how many share a row. A loader that ignores the sidecars and reads the .bin files as one flat stream will not respect document boundaries.

Sources and mix rationale

• —pg19: full public-domain books (emozilla/pg19, a parquet mirror of deepmind/pg19 -- the original is a script-based dataset repo that hits a gzip-handling regression in current datasets versions). The only source with documents comfortably longer than 8192 tokens; given the largest allocation for that reason.
• —arxiv: togethercomputer/RedPajama-Data-1T, arxiv subset.
• —wiki: wikimedia/wikipedia (20231101.en), filtered to min_tokens=6000 (raised from an initial 2000). Wikipedia articles are structurally short -- even the longest ones rarely approach 8192 tokens -- so this source is intentionally small and serves as a distribution anchor, not primary long-context material.
• —edu_long: HuggingFaceFW/fineweb-edu (sample-10BT), length-filtered.
• —stack_python: HuggingFaceCode/stack-v3-train, grouped to repository-level documents (files concatenated in priority order: setup/config files, then __init__.py, then other .py, then tests), Python-majority repos only. Capped at `--max-doc-tokens 65536`: an uncapped build showed the top 5 repositories (of ~15,670) accounting for 27.9% of all stack_python tokens, with the single largest single document (60.75M tokens) being ~96% ns-3 pybindgen auto-generated Python-bindings boilerplate -- not organic code. After the cap, top-5 concentration is 0.07%.
• —replay: a sample of the original 100B-token pretraining mix (harims95/hobbylm-mix100b-gpt2, dclm/code/math/anneal families). The original per-document boundaries were never recorded upstream -- prepare_mix100B.py only preserves them as an inline EOT token prefixed to each document in the flat stream. An earlier version of this corpus treated each 8M-token raw-shard slice as one "document" for indexing purposes; besides being far too coarse (median real document length is 698 tokens, not 8,000,000 -- see below), the metadata key used to describe that slice's position collided with and silently corrupted the shard-relative offset the loader actually needs, so 36 of 39 slices pointed at out-of-range byte ranges. Both are fixed: scripts/repair_replay_index.py reconstructs real per-document boundaries from the inline EOT markers (scripts/build_long_context_corpus.py's split_by_eot()), giving replay the same real-document-level indexing as every other source.

The mix was rebalanced from an initial plan (pg19 400M, arxiv 600M, wiki 350M, edulong 350M) after the initial wiki build showed 53.8% padding waste at seqlen 8192 under one-document-per-row training. That waste isn't wiki-specific -- any source with a median length under 8192 has the same problem, arxiv included, just less severely (its median is 15-25K tokens, PG19 is the only source with a genuinely comfortable margin). Two changes followed: short sources were cut and long sources increased (see table above), and block-diagonal document packing was implemented in the training loader itself so padding waste is no longer coupled to per-source document length at all.

Padding waste: before and after packing

The _docs.jsonl per-document lengths above imply real waste under a naive one-document-per-row loader (padding each doc out to a fixed row):

replay's real per-document lengths (median 698 tokens) make it, once correctly indexed, the single worst-case source for one-doc-per-row waste in this corpus -- worse than wiki. It's also the clearest illustration of why the packing loader matters: with it, the same fine-grained document boundaries that make one-doc-per-row training wasteful for replay cost nothing.

With boundary_aware_data_generator's packing (documents concatenated continuously into each row, one incomplete tail row per epoch per rank is the only possible waste), projected waste is negligible regardless of source or seqlen -- on the order of `(seqlen - 1) / total_tokens`:

Related

• —Model: harims95/hobbylm-1b-hf
• —Original pretraining mix: harims95/hobbylm-mix100b-gpt2
• —Routing dynamics across pretraining: harims95/hobbylm-routing-dynamics