Datasetpublic

JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids

luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by gpt-5.6-luna. Each thought is visible reasoning about the next 8 Qwen3 tokens after a cut, written without ever seeing that continuation. Intended to be spliced into the document before the chunk so a small model (Qwen3-4B-Base) can read the reasoning and predict the chunk. Thoughts are strings, not token ids;… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.

sourceHugging Facecc-by-4.0updated 28d agoView on Hugging Face

0likes119downloads

Dataset Card

luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids

Thoughts for next-token prediction on k=8 token chunks of `JackHsieh/statML-arxiv-40M-20M`, generated by gpt-5.6-luna. Each thought is visible reasoning about the next 8 Qwen3 tokens after a cut, written without ever seeing that continuation. Intended to be spliced into the document before the chunk so a small model (Qwen3-4B-Base) can read the reasoning and predict the chunk. Thoughts are strings, not token ids; tokenized tag-wrapped variants are separate datasets (.tags, .kv-tags, .kv-tags-explained suffixes).

Qwen3 counterpart of `luna-reason-only.k-8.statml-arxiv-llama32`. The two share document uuids and chunk-index grids but not cut positions: 4096 Qwen3 tokens span slightly different text than 4096 Llama-3.2 tokens, so chunk k falls at a different character offset in each. Thoughts are not interchangeable between them.

Note that input_tokens, cached_tokens, output_tokens columns measure the OpenAI tokens billed for this request, not Qwen3 tokens; kept for cost auditing |

Generation

Model	[gpt-5.6-luna](https://developers.openai.com/api/docs/models/gpt-5.6-luna)
Sampling parameters	Responses API defaults (no sampling parameters set); `reasoning.effort = "none"`, so there is no hidden reasoning
Stopping rule	Stops naturally or at the max output length of 450 generator tokens
Inputted prefix	Last ≤ 1024 Qwen3 tokens of the prefix, ending exactly at the cut
Generator prompt	`v5trim-qwen` (`luna_trim_qwen.py`) — the Llama run's `v5trim` retargeted to the Qwen3 tokenizer and reader, with Qwen3 length exemplars and Qwen3-cut few-shot examples

Thought lengths in Qwen3 tokens

split	mean ± std	[min, max]	truncated % (cap @ 450 generator tokens)
`train`	308.2 ± 56.5	[72, 769]	1.32%
`test`	309.8 ± 56.2	[103, 540]	1.36%
combined	308.4 ± 56.5	[72, 769]	1.33%

Generation metrics

Generation resources	OpenAI Batch API (622 refusal-recovery chunks re-sent synchronously)
Throughput	?
Total time for train	?
Total time for test	?
API cost	≈ $282 computed from the dataset's billing columns at published Batch-tier pricing (input $0.20 / cached $0.02 / output $1.20 per 1M tokens, 50% off standard); not an invoice figure
Moderation refusals	~4% of requests refused as `invalid_prompt` on benign LaTeX; stochastic, all recovered within 4 identical retries (85% on the first) — see below

Moderation refusals

~4% of requests were refused as invalid_prompt on entirely benign LaTeX — FDR tables, accuracy tables, convergence bounds. These are stochastic, not content-determined: identical retries recovered them at high rates, and all 622 residual chunks succeeded within 4 identical synchronous attempts (85% on the first). No prompt in this corpus is deterministically refused. Retries re-send the prompt unchanged; the prefix end, which defines the prediction task, is never moved.

Known characteristics

—~70% of thoughts open with the words "The cut is" — a stable habit of the generator that no prompt variant dislodged. Anything distilled from these will inherit it.
—Early chunks see a shorter prefix: chunk_index=8 has only 64 Qwen3 tokens of context, so those thoughts are competent but under-informed relative to full-window ones.

Coverage details

Documents are 4096 Qwen3 tokens = 512 chunks at chunk size k=8; the first chunk of a document is never thoughtful.

Split	Stride	Thoughts covered / doc	Total thinking chunks	`g`	Total thoughts
`train`	8	63 / doc	612_864	1	612_864
`test`	32	15 / doc	72_960	1	72_960
combined			685_824		685_824

Coverage is complete: every designated chunk has a thought.