CoolFace
Datasetpublic

JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled

luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled Every non-first chunk of every document carries a thought: the gpt-5.6-luna reasoning thought where one was generated, and a content-free pause thought everywhere else. luna chunk <|reserved_special_token_1|> luna reasoning <|reserved_special_token_2|> filler chunk <|reserved_special_token_1|> 256x <|reserved_special_token_0|> <|reserved_special_token_2|> The filler is 258 tokens. Chunk 0 is excluded… See the full description on the dataset page: https://huggingface.co/datasets/JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face
0likes118downloads
Dataset Card

luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags-pausefilled

Every non-first chunk of every document carries a thought: the gpt-5.6-luna reasoning thought where one was generated, and a content-free pause thought everywhere else.

luna chunk <|reservedspecialtoken1|> luna reasoning <|reservedspecialtoken2|> filler chunk <|reservedspecialtoken1|> 256x <|reservedspecialtoken0|> <|reservedspecialtoken_2|>

The filler is 258 tokens. Chunk 0 is excluded (empty prefix), so each 4096-token document contributes indices 1..511.

splitlunapause-fillertotal
train612,8644,358,1444,971,008
test72,9602,412,5442,485,504

Use with the stride=1,k=8 chunking sidecar (r = 511/512 = 1.000), which marks every non-first chunk thoughtful.

Provenance

Luna rows are copied unchanged from `JackHsieh/luna-reason-only.k-8.statml-arxiv-llama32.llama32-ids.tags` — prefix-only reasoning about the next 8 Llama-3.2 tokens, generated by gpt-5.6-luna at reasoning.effort="none". Filler rows are synthetic. Documents: JackHsieh/statML-arxiv-40M-20M-llama32. Tokenizer: meta-llama/Llama-3.2-3B; tag ids are inserted explicitly, so no BOS is introduced.

Schema

Source columns, plus:

columnnotes
input_idsLlama 3.2 ids of the wrapped thought, tags included
thought_textthe wrapped thought as text (large_string: the filler repeats 6,770,688 times)
gwithin-work-group index; always 0
thought_sourceluna (685,824 rows) or pause-filler (6,770,688 rows)

Training note

All three reserved tokens must be re-initialized in trainee.special_tokens (init: gaussian-diag). In Llama-3.2 every <|reserved_special_token_N|> row is the SAME vector, so untouched, the open tag, pause token and close tag would be indistinguishable and the filler would carry no positional information at all.