CoolFace
Datasetpublic

JackHsieh/4B-ranked-v9b.stride-train2-test32.k-8.statml-arxiv.qwen3-ids

4B-ranked-v9b.stride-train2-test32.k-8.statml-arxiv.qwen3-ids Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by Qwen/Qwen3-4B in thinking mode. The prompted task is not "write a thought". Instead, it is predict the next k=8 tokens after the cut, and report the prediction as a ranked list of 3–6 candidate continuations, each one being the tail copied verbatim plus the predicted continuation, written as N. «tail + continuation».… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/4B-ranked-v9b.stride-train2-test32.k-8.statml-arxiv.qwen3-ids.

sourceHugging Faceupdated 27d agoView on Hugging Face
0likes55downloads
Dataset Card

4B-ranked-v9b.stride-train2-test32.k-8.statml-arxiv.qwen3-ids

Thoughts for next-token prediction on k=8 token chunks of `JackHsieh/statML-arxiv-40M-20M`, generated by `Qwen/Qwen3-4B` in thinking mode.

The prompted task is not "write a thought". Instead, it is predict the next k=8 tokens after the cut, and report the prediction as a ranked list of 3–6 candidate continuations, each one being the tail copied verbatim plus the predicted continuation, written as N. «tail + continuation». Both the <think> block and the answer are stored in this dataset.

Record layout

input_ids is the assistant turn in full, including the <think> block prefill, the rest of the <think> generation, and the post-<think> answer.

<think>\nThe tail is «…». The tail ends with these characters: «…».\n\nLooking at the text:   ← prefill (31–167 tokens)
  … the model's reasoning …                                                                   ← generated
</think>
1. «tail + continuation»                                                                      ← generated
2. «tail + continuation»
3. «tail + continuation»

The <think> block is prefilled with the tail and seam quoted back to the model.

  • —n_prefill_tokens records where the prefill ends and generation begins.
  • —has_think_end records whether the model actually emitted </think> (which is true for 98.77% of rows in both splits).

Generation

Model[Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B), thinking mode
Sampling parametersQwen3 thinking-mode recommendation: temperature=0.6, top_p=0.95, top_k=20, min_p=0, plus presence_penalty=1.0, repetition_penalty=1.0
Stopping ruleStops when outputting `<\im_end\>` (token id 151645) or reaches the max completion length of 2048 tokens
Inputted prefixFull document prefix before the cut (≤ 4096 tokens)
Quoted tailLine-anchored (tail_mode=line): starts at the first visible character of a line, widened back a line at a time to ≥ 8 tokens, capped at 64 tokens with a re-cut at a sentence then word boundary. Always an exact suffix of the prefix, so "copy the tail verbatim" is well defined
Generator chat template`post-think-ranked-v9b.jinja`, which names the task, the five conditions, one paragraph on how to think, the output format, and three answer-only exemplars
vllm.max_model_len8192

Stored sequence lengths in Qwen3 tokens

Lengths below are of the whole stored sequence (prefill + completion). The <think> block prefill is 31–167 tokens it quotes the tail, whose length varies). truncated % is the completion hitting the 2048 cap, which is why max slightly exceeds 2048.

**split****mean ± std****[min, max]**truncated % (completion cap @ 2048)
train617.9 ± 314.6[67, 2205]1.5916%
test619.3 ± 312.8[179, 2186]1.5803%
combined617.9 ± 314.6[67, 2205]1.5914%

Format compliance

The chat template provides five conditions that the post-<think> answer should satisfy. On a 2000-thought subsample of each test, hard-coded classification by these conditions yields:

outcomeshare
fully compliant (3–6 distinct candidates, each the tail verbatim plus a real extension)~70%
all candidates identical (condition 5)13%
other deviations (mostly a delimiter closed where the tail cut mid-token)10%
candidates start at the tail's last line only (condition 1)2%
tail repeated with no extension4%
no </think> or no list at all1%

<!-- Two caveats. Part of the "other deviations" bucket is materially fine — the model closes a math delimiter the tail cut mid-token and then extends normally — so real compliance is nearer 75–80%. And the duplicate-candidate failures concentrate on cuts landing inside LaTeX tables and equations, where the continuation is close to deterministic and there genuinely aren't 3 distinct plausible candidates; test's stride-32 sampling hits those regions at a different rate than train's stride-2. Budget for ~20–25% of rows being unusable for a task that needs a well-formed ranked list. The kv variant's hint_failed flag catches only the harder failures (~4.5%), not condition-5 duplicates. -->

Generation metrics

Generation resources8 × NVIDIA H100 80GB HBM3
Throughput (train + test combined)≈ 36_049 tokens / s<br>~58 thoughts / s
Total tokens generated3110724_463
Total time for train≈ 23.4 hours (generation wall clock; single batch window, no preemption)
Total time for test≈ 0.6 hours (36 min)

Coverage details

Documents are 4096 Qwen3 tokens = 512 chunks at chunk size k=8; the first chunk of a document is never thoughtful.

**Split****Docs****Stride****Thoughts covered / doc****Total thinking chunks****`g`****Total thoughts**
train97282255 / doc248064024961280
test48643215 / doc72_960172_960
combined2_553_6005_034_240

<!--

Companion dataset

`…qwen3-ids.kv-tags-explained` is the same 5034240 rows rewritten as key/value hint tags for the rewrite-RL setup: the post-`</think>` ranked list only (capped at 512 tokens) wrapped as <|note|> … KEY … VALUE: … <|/note|>, with the reasoning discarded. Rows whose thought gave no usable answer — no </think>, empty, over-long, or a candidate list that never extends past the tail — carry the fixed text No hint was produced for this key. with hint_failed=True and a failure_reason. Row order and keys align 1:1 with this dataset. -->