JackHsieh/statML-arxiv-42M-11M-L-1024-llama32-pause7-injected

A pause-injected variant of JackHsieh/statML-arxiv-42M-11M-L-1024-llama32: every document token is preceded by a sentinel-wrapped stretch of pause tokens, so each row is exactly 8x longer (1024 -> 8192 tokens). Nothing else changes — same papers, same windows, same train/test split, same schema and column order as the parent.

How it was derived

For each row of the parent, the token sequence [t0, t1, ...] becomes:

<|reserved_special_token_1|> <|reserved_special_token_0|> x5 <|reserved_special_token_2|> t0
<|reserved_special_token_1|> <|reserved_special_token_0|> x5 <|reserved_special_token_2|> t1
...

Each group is 8 tokens — 7 pause-stretch tokens then the one real token — so under a k=8 chunking every chunk lands exactly on one group, and a token's pause stretch is precisely the chunk prefix that predicts it. These three are existing reserved tokens in the meta-llama/Llama-3.2-3B vocab, so no resize is needed; note they are untrained placeholders that all share one embedding direction at initialization, so a trainee must re-initialize each of the three rows separately.

Schema

Identical to the parent, including column order. Two columns change value:

• —input_ids — the interleaved sequence, exactly 8_192 int32 ids. Authoritative. Verified on every row: each group is [stretch, real token], and the real tokens in order reproduce the parent row exactly.
• —text — the decode of input_ids with every sentinel spelled out.
• —Llama3.2_token_count — set to the new row length (8192), i.e. `len(inputids)`.

text fidelity caveat (affects 454 of 51_200 rows, 0.89%)

tokenizer.encode(text, add_special_tokens=False) == input_ids holds for 99.11% of rows, but not all — use input_ids, never a re-tokenization of text, as the source of truth. 108 distinct token ids in the corpus do not survive being decoded in isolation, which is what putting a sentinel between every pair of tokens forces:

• —Partial-UTF-8 byte tokens (the lossy case). A single token can hold an incomplete UTF-8 byte sequence — fine in the parent, where neighbouring tokens complete the character. Separated by sentinels, each fragment decodes to U+FFFD (�) and the original bytes are unrecoverable from `text`. input_ids still holds the exact original ids, so nothing is lost from the data itself.
• —Punctuation-variant ids (cosmetic only). The vocab has duplicate surface forms — e.g. id 662 and id 13 both decode to ".", as do "," (1174 / 11) and "..." (2564 / 1131). Encoding that surface alone returns the canonical id, so a re-tokenization of text yields the twin id. The rendered characters are identical and correct.

Rows are unaffected unless they contain one of those ids; the affected uuids are all in documents with non-Latin script or unusual punctuation runs.

uuid, start_index (the window's offset in the source paper's token sequence), and the paper metadata are carried over untouched, so rows still join to the parent by uuid.

Size warning. Each sentinel spells out as 28 characters, so text is ~64x larger than the parent's (~204k chars/row). Parquet compresses the repetition heavily here, but load_dataset materializes uncompressed Arrow locally — budget ~11 GB.

Built by prestar_prep/subset/inject_pause_stretches.py.