datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by Qwen3-32B
with thinking mode off. Each thought is a few dense sentences of reasoning about the next
8 tokens after a cut, written from the document prefix alone — the generator never sees the
continuation. Stored thought_text includes the <thought>/</thought> wrapper.
The 4B parity counterpart is… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
Qwen3-4B-Instruct-2507. Each thought is a few dense sentences of reasoning about the next
8 tokens after a cut, written from the document prefix alone — the generator never sees the
continuation. Stored thought_text includes the <thought>/</thought> wrapper.
This is the small-generator parity counterpart of… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/4B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
Qwen3-4B-Instruct-2507.prestar-RL.reason-only.lr7e-7-kl0.step-2176 — Qwen3-4B-Instruct-2507 after RL against a frozen suffix conditional. Each thought is a few dense sentences of reasoning about the next 8 tokens after a cut, written from the document prefix alone — the generator never sees the… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
gpt-5.6-luna. Each thought is visible reasoning about the next 8 Qwen3 tokens after a cut,
written without ever seeing that continuation. Intended to be spliced into the document
before the chunk so a small model (Qwen3-4B-Base) can read the reasoning and predict the
chunk. Thoughts are strings, not token ids;… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled
Every non-first chunk of every document carries a thought: the gpt-5.6-luna reasoning thought
where one was generated, and a content-free pause thought everywhere else.
luna chunk <|reserved_special_token_1|> luna reasoning <|reserved_special_token_2|>
filler chunk <|reserved_special_token_1|> 256x <|reserved_special_token_0|> <|reserved_special_token_2|>
The filler is 258 tokens. Chunk 0 is excluded… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled.4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids
4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids
Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by
qwen3-4B-instruct-luna-distill-step2362
This generator is Qwen3-4B-Instruct-2507 SFT'd on the train set of gpt-5.6-luna thoughts (wandb run 3rkrz2vo, step 2362).
⚠️ Note that stride-8 chunks were also seen during training this generator. There is a concern that the distillation checkpoint may have memorized… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.tags
4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.tags
Tokenized, tag-wrapped form of JackHsieh/4B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as <|note|><thought><|/note|> and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base token ids (input_ids).
Longest thought: 510 tokens — a training run's max_thought_length must be at least
this.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670.
A run must declare these under… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.tags.4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-distill-3rkrz2vo.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A length-matched null control for JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids, stored directly in the
kv-tags-explained training format (there is no separate base repo). Every thought is this
one sentence, repeated:
We are thinking hard about what comes in the next 8 tokens by reasoning correctly and carefully about what comes before it in the document.
It is fluent, on-topic and… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv
4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv
Reasoning about the next 8 Qwen3 tokens of stat.ML arXiv LaTeX, generated by
stock Qwen/Qwen3-4B-Instruct-2507 -- no fine-tuning, prompted with the reason-only-nothink.jinja reasoning template.
Documents: JackHsieh/statML-arxiv-40M-20M, 4096 Qwen3 tokens each.
Why this checkpoint: Stock Qwen3-4B-Instruct-2507 prompted with reason-only-nothink.jinja, a byte-identical copy of the reason-only.jinja template used for… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.tags
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.tags
A pre-tokenized, tag-wrapped variant of JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>{thought_text}<|/note|>
and stored both as text (thought_text) and as Qwen/Qwen3-4B-Base token ids (input_ids).
Tag ids and the spliced key are inserted as ids, never re-tokenized.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670. These are the first… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.tags.luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags
A pre-tokenized, tag-wrapped variant of JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text (thought_text) and as Qwen/Qwen3-4B-Base token ids (input_ids).
Tag ids and the spliced key are inserted as ids, never re-tokenized.
Delimiter ids: <|note|> = 151669… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.kv-tags.4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids, the thoughts written by the prestar-RL policy
Qwen3-4B-Instruct-2507.prestar-RL.reason-only.lr7e-7-kl0.step-2176.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-Instruct-RL-step2176-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/8B-reason-only.stride-32-test.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.luna-reason-only.k-8.statml-arxiv-llama32
luna-reason-only.k-8.statml-arxiv-llama32
Prefix-only "thoughts" for next-token prediction on stat.ML arXiv LaTeX. Each thought is
visible reasoning about the next 8 Llama-3.2 tokens after a cut, written without ever
seeing that continuation. Intended to be spliced into the document before the chunk so a
small model (Llama 3.2 3B base) can read the reasoning and predict the chunk.
Complete: every designated chunk has a thought.
split
thoughts
coverage
documents… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.qwen3-distill-3rkrz2vo.k-8.statml-arxiv-qwen3.qwen3-ids.tags
qwen3-distill-3rkrz2vo.k-8.statml-arxiv-qwen3.qwen3-ids.tags
Tokenized, tag-wrapped form of JackHsieh/qwen3-4B-instruct-luna-distill-step2362.reason-only.k-8.statml-arxiv-qwen3.
Each thought is wrapped as <|note|> … thought … <|/note|> and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base token ids (input_ids).
Longest thought: 770 tokens — a training run's max_thought_length must be at least
this.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670.
A run must… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/qwen3-distill-3rkrz2vo.k-8.statml-arxiv-qwen3.qwen3-ids.tags.32B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
32B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/32B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained.32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained
A pre-tokenized, tag-wrapped variant of JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids. Each thought_text is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {thought_text}
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as <|reserved_special_token_1|> … thought … <|reserved_special_token_2|> … last prefix token and stored both as text (thought_text) and as Llama 3.2
token ids (input_ids).
The trailing token is the document token immediately before the cut (input_ids[chunk_start_index - 1]), copied from the document rather… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-echo.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as
<|reserved_special_token_1|>
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|reserved_special_token_2|>
and stored both as text (thought_text) and as Llama 3.2
token ids (input_ids).
Longest thought: 514 tokens — a training run's max_thought_length must be at least
this.
Intended to be PREPENDED to the… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags.qwen3-distill-1e5-1ep-s480-rec.k-8.statml-arxiv-qwen3
qwen3-distill-1e5-1ep-s480-rec.k-8.statml-arxiv-qwen3
Reasoning about the next 8 Qwen3 tokens of stat.ML arXiv LaTeX, generated by a
Qwen3-4B-Instruct-2507 distilled on gpt-5.6-luna thoughts
(SFT: lr 1e-5, batch 256, 1-epoch cosine; this is step 480, 0.2 epochs of data seen).
Documents: JackHsieh/statML-arxiv-40M-20M, 4096 Qwen3 tokens each.
Why this checkpoint: the most lightly tuned checkpoint, so its thoughts stay closest to the original Qwen3-4B-Instruct behaviour
Sampling:… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/qwen3-distill-1e5-1ep-s480-rec.k-8.statml-arxiv-qwen3.qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3.qwen3-ids.kv-tags-explained
qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3.qwen3-ids.kv-tags-explained.luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags-explained
luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.
Each thought is wrapped as
<|reserved_special_token_1|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|reserved_special_token_2|>
and stored both as text (thought_text)… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.kv-tags-explained.qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3
qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3
Reasoning about the next 8 Qwen3 tokens of stat.ML arXiv LaTeX, generated by a
Qwen3-4B-Instruct-2507 distilled on gpt-5.6-luna thoughts
(SFT: lr 3e-5, batch 256, 4-epoch cosine; this is step 4724, 2.0 epochs of data seen).
Documents: JackHsieh/statML-arxiv-40M-20M, 4096 Qwen3 tokens each.
Why this checkpoint: the best held-out perplexity on luna thoughts across the lr sweep
Sampling: greedy (temperature 0), seed 0, one… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/qwen3-distill-3e5-4ep-s4724-greedy.k-8.statml-arxiv-qwen3.mantra-offset4-lenmatched.k-8.statml-arxiv-qwen3
mantra-offset4-lenmatched.k-8.statml-arxiv-qwen3
A length-matched null control for JackHsieh/luna-reason-only.k-8.statml-arxiv-qwen3.
Every thought is this one sentence, repeated:
We are thinking hard about what comes in the next 8 tokens by reasoning correctly and carefully about what comes before it in the document.
It is fluent, on-topic and completely uninformative — it says nothing about the document it
sits in. Paired against luna at the same cuts, it isolates whether… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/mantra-offset4-lenmatched.k-8.statml-arxiv-qwen3.4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text)… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-reason-only.stride-train8off4-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained.32B-reason-only.stride-train8-test32.k-8.L-512.statml-arxiv.qwen3-ids.tags
32B-reason-only.stride-train8-test32.k-8.L-512.statml-arxiv.qwen3-ids.tags
Tokenized, tag-wrapped form of JackHsieh/32B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as <|note|> … thought … <|/note|> and stored both as text (thought_text) and as
Qwen/Qwen3-4B-Base token ids (input_ids).
Longest thought: 509 tokens — a training run's max_thought_length must be at least
this.
Delimiter ids: <|note|> = 151669, <|/note|> = 151670.
A run must declare these… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-train8-test32.k-8.L-512.statml-arxiv.qwen3-ids.tags.32B-reason-only.stride-train8-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
32B-reason-only.stride-train8-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained
Tokenized, tag-wrapped form of JackHsieh/32B-reason-only.rule-r-1.0-k-8.L-512.statml-arxiv.
Each thought is wrapped as
<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: <last 8 prefix tokens>
VALUE: <thought>
<|/note|>
and stored both as text (thought_text) and as… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-train8-test32.k-8.L-512.statml-arxiv.qwen3-ids.kv-tags-explained.