ryan-0608/MoS-Qwen3-8B-EAGLE3-responses

MoS — Qwen3-8B EAGLE3 Training Responses

Target-model responses for training EAGLE3 speculative-decoding draft models against [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B). Built for the MoS (Mixture of Speculators) project — a routed multi-MLP draft — and equally usable for any single-draft EAGLE3 / SpecForge training run on Qwen3-8B.

599,087 complete assistant responses (with thinking traces) over five domains, generated by Qwen3-8B itself so the draft learns to mimic the target's own distribution.

Per-domain counts

Schema

Each line is one JSON record:

{
  "id": "<stable id>",
  "conversations": [
    {"role": "user", "content": "<prompt>"},
    {"role": "assistant", "content": "<final answer>", "reasoning_content": "<thinking trace>"}
  ],
  "domain_id": 0
}

• —reasoning_content holds the model's <think> trace (thinking mode was enabled); content holds the post-thinking answer. Some records have an empty content when the response was entirely thinking.
• —domain_id is the integer above; it is the ground-truth routing label used by MoS (router supervision in stage 2, teacher-forced expert routing in stage 3).

Generation

• —Target: Qwen/Qwen3-8B, bf16, vLLM.
• —Decoding: greedy, thinking enabled.
• —Length: generated complete to EOS with max_tokens = max_model_len = 16384, then truncated at training time (max_length = 4096) — this mirrors SpecForge's approach (store the full response, truncate when building the training batch) and avoids the mid-thought truncation that corrupts a "generate-to-a-short-cap" pipeline.
• —Long greedy outputs are kept verbatim — no synthetic repetition filtering. Greedy degeneration is rare here: only 8 math records (0.01%) exceed 80K characters, the longest being 83,450 characters (still within the 16,384-token cap; Qwen BPE packs repeated runs densely).

Train/eval decontamination

Decontamination is done on prompts, before generation, so the training set is disjoint from evaluation by construction:

1. 1.Normalize each user prompt and take its md5.
2. 2.Drop any train prompt whose hash collides with an eval prompt (eval leakage).
3. 3.Drop intra-train duplicate prompts (keep one).
4. 4.Drop empty prompts.

619,177 → 599,087 (eval-leak 744 + intra-train duplicate 19,345 + empty 1 = 20,090 removed). The merged corpus is then re-verified: 0 malformed JSON, 0 byte-wrapped prompts, 0 empty prompts, 0 duplicate ids, all schema-valid.

Usage

from datasets import load_dataset

ds = load_dataset("ryan-0608/MoS-Qwen3-8B-EAGLE3-responses", split="train")
print(ds[0]["conversations"][0]["content"])      # user prompt
print(ds[0]["conversations"][1]["reasoning_content"])  # thinking trace
print(ds[0]["domain_id"])                         # routing label

Load a single domain:

ds_math = load_dataset(
    "ryan-0608/MoS-Qwen3-8B-EAGLE3-responses",
    data_files="data/math_train.jsonl", split="train",
)

License

Apache-2.0, following the Qwen3 license. The responses are generated by Qwen/Qwen3-8B; the source prompts are drawn from public instruction/reasoning datasets.