Datasetpublic

JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32

luna-reason-only.k-8.statml-arxiv-llama32 Prefix-only "thoughts" for next-token prediction on stat.ML arXiv LaTeX. Each thought is visible reasoning about the next 8 Llama-3.2 tokens after a cut, written without ever seeing that continuation. Intended to be spliced into the document before the chunk so a small model (Llama 3.2 3B base) can read the reasoning and predict the chunk. Complete: every designated chunk has a thought. split thoughts coverage documents… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes35downloads

Dataset Card

luna-reason-only.k-8.statml-arxiv-llama32

Prefix-only "thoughts" for next-token prediction on stat.ML arXiv LaTeX. Each thought is visible reasoning about the next 8 Llama-3.2 tokens after a cut, written without ever seeing that continuation. Intended to be spliced into the document before the chunk so a small model (Llama 3.2 3B base) can read the reasoning and predict the chunk.

Complete: every designated chunk has a thought.

split	thoughts	coverage	documents	thoughts/doc
`train`	612,864	100.00%	9,728 / 9,728	63 / 63
`test`	72,960	100.00%	4,864 / 4,864	15 / 15

Provenance


documents	`JackHsieh/statML-arxiv-40M-20M-llama32`, 4096 Llama-3.2 tokens/doc
chunking	`train`: `stride=8,k=8` — chunk indices 8, 16, …, 504 (r = 63/512 = 0.1230)<br>`test`: `stride=32,k=8` — chunk indices 32, 64, …, 480 (r = 15/512 = 0.0293)
generator	`gpt-5.6-luna`, Responses API, `reasoning.effort = "none"` (no hidden reasoning)
generator input	the last ≤1024 Llama tokens before the cut, ending exactly at the cut; never the continuation
template	`v5trim` — instructions + 2 worked examples as chat turns
output cap	450 generator tokens

Generated almost entirely via the Batch API; 14 chunks were finished with byte-identical synchronous requests after repeated batch refusals (see Moderation refusals).

Schema

One row per generated thought.

column	notes
`uuid`	document id; joins to the documents dataset above
`chunk_index`	0–511, the chunk this thought precedes
`chunk_start_index`	first token the thought is meant to help predict (the cut)
`chunk_end_index`	`chunk_start_index + 8`
`thought_text`	raw generator output, unmodified, no delimiters or tags
`chars`	length of `thought_text`
`llama_tokens`	`thought_text` under the Llama 3.2 tokenizer, `add_special_tokens=False`
`prefix_window_llama`	tokens of prefix the generator saw (< 1024 only for early chunks)
`truncated`	hit the 450-token output cap
`model`, `template_version`, `reasoning_effort`	generation provenance
`input_tokens`, `cached_tokens`, `output_tokens`	OpenAI tokens billed for this request, not Llama tokens; kept for cost auditing

Thoughts are strings, not token ids. Tokenized, tag-wrapped variants are published separately (.llama32-ids.tags, .llama32-ids.tags-echo).

Statistics

	train	test
Llama tokens/thought	mean 299, median 298, max 498	mean 300, median 299, max 482
chars/thought	mean 1307	mean 1310
`truncated` at the 450-token cap	4,492	524
> 384 Llama tokens	35,918 (5.9%)	4,389 (6.0%)

Consumers enforcing a length ceiling should trim at a sentence boundary rather than mid-clause.

Moderation refusals

The API refused ~4% of requests as invalid_prompt on entirely benign LaTeX — FDR tables, accuracy tables, convergence bounds. These refusals are stochastic, not content-determined: identical retries recovered 90%, 81%, 75%, 66% across successive batch rounds, and the final 14 survivors all succeeded within 4 identical synchronous attempts (6 of them on the first). No prompt in this corpus is deterministically refused. Retries always re-sent the prompt unchanged; the prefix end, which defines the prediction task, was never moved.

Known characteristics

—~70% of thoughts open with the words "The cut is" — a stable habit of the generator that no prompt variant dislodged. Anything distilled from these will inherit it.
—Early chunks see a shorter prefix: chunk_index=8 has only 64 tokens of context, so those thoughts are competent but under-informed relative to full-window ones.