JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids
luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids Thoughts for next-token prediction on k=8 token chunks of JackHsieh/statML-arxiv-40M-20M, generated by gpt-5.6-luna. Each thought is visible reasoning about the next 8 Qwen3 tokens after a cut, written without ever seeing that continuation. Intended to be spliced into the document before the chunk so a small model (Qwen3-4B-Base) can read the reasoning and predict the chunk. Thoughts are strings, not token ids;… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids.
Update README.md
Add luna_trim_qwen.py (generation provenance)
Canonicalize dataset card
Card: dataset complete at 100%; refusals shown stochastic, none deterministic
Test split COMPLETE: 72,960/72,960 chunks (100%), all 4,864 documents at 15/15
Train split COMPLETE: 612,864/612,864 chunks (100%), all 9,728 documents at 63/63
Update card: both splits, statistics, gap taxonomy
Add train split: 612,316 of 612,864 chunks (99.91%), all 9,728 documents; stride=8,k=8 on the Qwen3 corpus
Re-export with the correct prefix_window_qwen3 column name
Remove the 8-row pilot artifact: its schema differed, which made every split fail to load
Add dataset card: Qwen3 counterpart, provenance, schema, length caveat
Test split: 72,886 of 72,960 chunks (99.90%), all 4,864 documents; stride=32,k=8 on the Qwen3 corpus
PILOT ONLY (8 rows): verify chunk matching for the Qwen3 corpus before the full run
initial commit