Datasetpublic

JackHsieh/4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids

4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by Qwen/Qwen3-4B in thinking mode. The prompted task: reason about what comes IMMEDIATELY next — the next k=8 tokens after the cut — and answer with a single unconstrained paragraph of dense reasoning, focused on the exact state at the cut and what the local grammar, notation, or argument forces next. Both the… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.

sourceHugging Faceupdated 21d agoView on Hugging Face

0likes520downloads

Dataset Card

4B-think-paragraph.stride-train1-test32.k-8.statml-arxiv.qwen3-ids

Thoughts for next-token prediction on k=8 token chunks of `JackHsieh/statML-arxiv-40M-20M`, generated by `Qwen/Qwen3-4B` in thinking mode.

The prompted task: reason about what comes IMMEDIATELY next — the next k=8 tokens after the cut — and answer with a single unconstrained paragraph of dense reasoning, focused on the exact state at the cut and what the local grammar, notation, or argument forces next. Both the <think> block and the answer are stored in this dataset. Unlike the bare sibling, the answer carries no <thought> tags. Identical to `4B-think-reason-only…` except for the answer-format instruction.

Record layout

input_ids is the assistant turn in full: the 2-token <think>\n prefill, the rest of the <think> generation, and the post-</think> answer.

—n_prefill_tokens records where the prefill ends and generation begins (always 2 here).
—has_think_end records whether the model actually emitted </think> (98.48% of train rows, 98.44% of test).

Generation

Model	[Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B), thinking mode
Sampling parameters	Qwen3 thinking-mode recommendation: `temperature=0.6`, `top_p=0.95`, `top_k=20`, `min_p=0`, plus `presence_penalty=0`, `repetition_penalty=1.0`
Stopping rule	Stops when outputting `<\	im_end\	>` (token id 151645) or reaches the max completion length of 4096 tokens
Inputted prefix	Full document prefix before the cut (≤ 4096 tokens)
Generator chat template	`paragraph_v3.jinja`, which names the task and the focus on the immediate continuation, and asks for a paragraph answer
`vllm.max_model_len`	8960

Stored sequence lengths in Qwen3 tokens

Lengths below are of the whole stored sequence (the 2-token prefill + completion). truncated % is the completion hitting the 4096 cap, which is why max slightly exceeds 4096.

split	mean ± std	[min, max]	truncated % (completion cap @ 4096)
`train`	844.6 ± 586.8	[17, 4098]	1.4086%
`test`	846.3 ± 592.4	[31, 4098]	1.4433%
combined	844.6 ± 586.8	[17, 4098]	1.4088%

Generation metrics

Generation resources	8 × NVIDIA H100 80GB HBM3
Throughput (train + test combined)	≈ 43_300 tokens / s<br>~51 thoughts / s<br><sub>estimate: total tokens over generation time; the train run's first window (timed out at the 2-day wall) contributes its Slurm elapsed minus startup</sub>
Total tokens generated	8458588_825
Total time for train	≈ 54.2 hours of GPU allocation (Slurm elapsed across two batch windows: 48.0 h to the wall at ~89%, then 6.1 h resumed from the shards; no preemption). Its in-job finalize hit a disk quota, so the dataset was finalized out of band, CPU-only, from the shards
Total time for test	≈ 0.7 hours (43 min; single batch window, no preemption)

Coverage details

Documents are 4096 Qwen3 tokens = 512 chunks at chunk size k=8; the first chunk of a document is never thoughtful.

Split	Docs	Stride	Thoughts covered / doc	Total thinking chunks	`g`	Total thoughts
`train`	9728	1	511 / doc	4971008	2	9942016
`test`	4864	32	15 / doc	72_960	1	72_960
combined				5_043_968		10_014_976