JackHsieh/spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained

spam.stride-1.k-8.statml-arxiv.qwen3-ids.kv-tags-explained

A length-matched null control for `JackHsieh/luna-reason-only.stride-train8-test32.k-8.statml-arxiv.qwen3-ids`, stored directly in the kv-tags-explained training format (there is no separate base repo). Every thought is this one sentence, repeated:

We are thinking hard about what comes in the next 8 tokens by reasoning correctly and carefully about what comes before it in the document.

It is fluent, on-topic and completely uninformative — it says nothing about the document it sits in. Paired against luna at the same cuts, it isolates whether the content of a thought matters, holding position, format and length fixed.

Construction

Each thought is wrapped as

<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {mantra}
<|/note|>

and stored both as text (thought_text) and as Qwen/Qwen3-4B-Base token ids (input_ids). Tag ids and the spliced key are inserted as ids, never re-tokenized.

Delimiter ids: <|note|> = 151669, <|/note|> = 151670. These are the first two special token ids under the Qwen3 tokenizer; a training run must declare them under trainee.special_tokens in this order.

Intended to be prepended to the document rather than interjected between prefix and suffix: the KEY names the span the VALUE is about. The KEY is the 8 document tokens ending at the cut (input_ids[chunk_start_index - 8 : chunk_start_index]).

Thought lengths in Qwen3 tokens (input_ids length, tags included)

Coverage details

Documents are 4096 Qwen3 tokens = 512 chunks at chunk size k=8; the first chunk of a document is never thoughtful.