Datasetpublic

JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids

32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by Qwen3-32B with thinking mode off. Each thought is a few dense sentences of reasoning about the next 8 tokens after a cut, written from the document prefix alone — the generator never sees the continuation. Stored thought_text includes the <thought>/</thought> wrapper. The 4B parity counterpart is… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids.

sourceHugging Faceapache-2.0updated 28d agoView on Hugging Face

0likes750downloads

Dataset Card

32B-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids

Thoughts for next-token prediction on k=8 token chunks of `JackHsieh/statML-arxiv-40M-20M`, generated by Qwen3-32B with thinking mode off. Each thought is a few dense sentences of reasoning about the next 8 tokens after a cut, written from the document prefix alone — the generator never sees the continuation. Stored thought_text includes the <thought>/</thought> wrapper.

The 4B parity counterpart is `JackHsieh/4B-Instruct-reason-only.stride-1.k-8.statml-arxiv.qwen3-ids`: same chunk grid, same prompt, same sampling, smaller generator.

Generation

Model	[Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B), thinking mode off
Sampling parameters	`temperature=0.6`, `top_p=0.9`, `repetition_penalty=1.0`, unseeded
Stopping rule	Stops when outputting `</thought>`, or at the 512-token cap on the stored thought (wrapper tags included)
Inputted prefix	Full document prefix before the cut (≤ 4096 tokens)
Generator chat template	`reason-only.jinja` — an empty `<think></think>` prefill switches thinking off on this hybrid model

Thought lengths in Qwen3 tokens

split	mean ± std	[min, max]	truncated % (cap @ 512)
`train`	166.5 ± 34.1	[13, 512]	0.0295%
`test`	166.4 ± 34.0	[14, 512]	0.0254%
combined	166.5 ± 34.1	[13, 512]	0.0287%

Generation metrics

Generation resources	8 × NVIDIA H200
Throughput (train + test combined)	23_404 tokens / s<br>141 thoughts / s
Total time for train	? (resumed across several sessions; the final session took 0.9 hours)
Total time for test	4.8 hours

Coverage details

Documents are 4096 Qwen3 tokens = 512 chunks at chunk size k=8; the first chunk of a document is never thoughtful.

Split	Stride	Thoughts covered / doc	Total thinking chunks	`g`	Total thoughts
`train`	1	511 / doc	4971008	2	9942016
`test`	1	511 / doc	2485504	1	2485504
combined			7_456_512		12_427_520

Schema

One row per (chunk, g): uuid, chunk_index, chunk_start_index, chunk_end_index, g, input_ids (generator-tokenizer ids of the wrapped thought), thought_text, truncated, finish_reason, generated_at, generation_seconds.