JackHsieh/4B-think-reason-only.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained

4B-think-reason-only.stride-train1-test32.k-8.statml-arxiv.qwen3-ids.kv-tags-explained

A pre-tokenized, tag-wrapped variant of `JackHsieh/4B-think-reason-only.stride-train1-test32.k-8.statml-arxiv.qwen3-ids`.

• —The source thoughts are two parts: a <think> block, then a few dense sentences that the generator wrapped in a literal <thought>…</thought> block (plain text, not special tokens). Only the part after </think> becomes the VALUE, with every `<thought>` / `</thought>` tag removed (and a </think> the model occasionally re-emitted as plain text ahead of the tag); the reasoning inside the think block is dropped. Tags were dropped at the id level wherever they occupied whole tokens (~98.5% of rows); where BPE had merged a tag boundary into a neighbouring token (e.g. .</), the VALUE was re-encoded from its tag-stripped text (~1.5%).
• —The VALUE is capped at 512 tokens.
• —1.62% of rows (train 1.6151%: 159,357 no_think_end, 1,188 too_long, 24 empty; test 1.6173%) have no usable answer: the completion never closed </think>, nothing followed it, or the answer ran past the cap. Those rows carry the text No hint was produced for this key. with hint_failed=True and a failure_reason (no_think_end, too_long, or empty). There is still exactly one row per source row, in the same order.

Each thought_text is wrapped as

<|note|>
This is a hint about a span that appears later in this document. KEY is the text immediately before that span; VALUE is a note about what might come next.
KEY: {last 8 prefix tokens}
VALUE: {post-</think> answer}
<|/note|>

and stored both as text (thought_text) and as Qwen/Qwen3-4B-Base token ids (input_ids). Tag ids and the spliced key are inserted as ids, never re-tokenized.

Delimiter ids: <|note|> = 151669, <|/note|> = 151670. These are the first two special token ids under the Qwen3 tokenizer; a training run must declare them under trainee.special_tokens in this order.

Intended to be prepended to the document rather than interjected between prefix and suffix: the KEY names the span the VALUE is about. The KEY is the 8 document tokens ending at the cut (input_ids[chunk_start_index - 8 : chunk_start_index]).

Thought lengths in Qwen3 tokens (input_ids length, tags included)