Datasetpublic

JackHsieh/statML-arxiv-40M-20M-olmo3

statML-arxiv-40M-20M-olmo3 An OLMo 3-tokenized re-windowing of JackHsieh/statML-arxiv-40M-20M (originally tokenized with Qwen3). Each row is a contiguous span of exactly 4096 tokens under the OLMo 3 tokenizer (allenai/Olmo-3-1025-7B, vocab 100 278), built to be as close as possible to the corresponding Qwen3 window of the same paper so the two corpora are comparable across tokenizers. The schema mirrors the ancestor dataset (leading columns), followed by re-windowing/difference… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/statML-arxiv-40M-20M-olmo3.

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

0likes23downloads

Dataset Card

statML-arxiv-40M-20M-olmo3

An OLMo 3-tokenized re-windowing of `JackHsieh/statML-arxiv-40M-20M` (originally tokenized with Qwen3). Each row is a contiguous span of exactly 4096 tokens under the OLMo 3 tokenizer (allenai/Olmo-3-1025-7B, vocab 100 278), built to be as close as possible to the corresponding Qwen3 window of the same paper so the two corpora are comparable across tokenizers. The schema mirrors the ancestor dataset (leading columns), followed by re-windowing/difference metrics.

—train: 9,728 sequences (39.8M tokens) · test: 4,864 sequences (19.9M tokens)
—Same papers (matched by uuid) and same train/test split as the Qwen3 version.

How it was built

Ancestor: the raw full-text papers in `JackHsieh/statML-arxiv`. For each row of the Qwen3 dataset:

1.Same start (character offset). The Qwen3 window's start (a token index) is mapped to a character offset in the paper; the OLMo window begins there. Since 4096 Qwen tokens ≠ 4096 OLMo tokens of text, the window end drifts. (start_index in this dataset is that character offset.)
2.Exactly 4096 OLMo tokens from the offset, with a decode→re-encode==4096 check (minimal end-nudge; a raw-ids fallback flagged by reencode_ok=False in the rare case it can't be met).
3.Shortfall → end-anchor. If the paper is too short after the offset, the window is shifted earlier and anchored to the paper's final 4096 OLMo tokens (anchor="end").
4.Whole paper < 4096 OLMo tokens → right-pad with <|extra_id_10|> (id 100 275) up to 4096. These fill tokens are genuine, non-masked tokens — a reserved sentinel chosen so it is never confused with <|pad|>. Padded rows have `end_pad_count > 0`.

Reproduce (tokenizer-agnostic; same script serves the planned Llama port):

bash

python -m prestar_prep.retokenize.rewindow_documents \
  --ancestor-tokenizer Qwen/Qwen3-4B-Instruct-2507 \
  --target-tokenizer allenai/Olmo-3-1025-7B \
  --fill-token "<|extra_id_10|>" --num-proc 16 --out-dir <out>

Schema

Leading columns (mirror the ancestor statML-arxiv-40M-20M):

column	type	meaning
`text`	str	decoded OLMo window text
`arxiv_id`, `url`, `timestamp`, `yymm`, `language`	str	per-paper metadata (joined by `uuid`)
`token_count`	int64	window length in tokens (4096, for schema parity)
`uuid`	str	paper id; matches the ancestor + Qwen3 datasets
`start_index`	int64	character offset in the paper where the window begins
`input_ids`	list[int]	the 4096 OLMo-3 token ids (what training consumes)

Auxiliary re-windowing metrics (appended):

column	meaning
`start_char` / `eff_start_char`	original vs. actual char-offset anchor (differ only when end-anchored)
`ancestor_start_index`	the original Qwen3 token `start_index`
`anchor`	`start` · `end` (shortfall) · `pad` (short paper)
`end_pad_count`	fill tokens appended (> 0 ⇒ padded; genuine, non-masked)
`ancestor_char_len` / `target_char_len`	char length of the Qwen vs OLMo window
`char_iou`, `end_drift`, `start_shift_chars`, `nudged`, `reencode_ok`	per-window difference/provenance

Difference from the Qwen3 windows

metric	train	test
char-span IoU (mean / min)	0.980 / 0.447	0.980 / 0.595
end-drift mean (chars)	271	275
end-anchored (shortfall)	1.54%	1.73%
end-padded (`end_pad_count>0`)	12	3
re-encode fallback	0.16%	0.19%

Windows sharing the same start with the Qwen3 corpus overlap ~98% of their character span; the OLMo window extends a few hundred characters further on average (OLMo needs slightly more text per 4096 tokens on this math-heavy content).