susun-123/perfectblend-qwen3-4b-regen
perfectblend-qwen3-4b-regen Single-turn SFT-style corpus for training speculative-decoding draft models (DFlash/MTP-style) against Qwen/Qwen3-4B as the target. Prompts come from an open-perfectblend-derived blend; every assistant response was regenerated by Qwen3-4B itself, so the token distribution matches the target model exactly. The sampled output_token_ids are included, letting trainers supervise on the target's own decode without re-tokenization drift.… See the full description on the dataset page: https://huggingface.co/datasets/susun-123/perfectblend-qwen3-4b-regen.
perfectblend-qwen3-4b-regen
Single-turn SFT-style corpus for training speculative-decoding draft models (DFlash/MTP-style) against Qwen/Qwen3-4B as the target. Prompts come from an open-perfectblend-derived blend; every assistant response was regenerated by Qwen3-4B itself, so the token distribution matches the target model exactly. The sampled output_token_ids are included, letting trainers supervise on the target's own decode without re-tokenization drift.
Generation
Decode fidelity was verified by teacher-forcing sampled rows through Qwen3-4B: 100% of output tokens fall inside the top-k=20 candidate set (max observed rank 10), 99.9% inside the top-p=0.8 nucleus, and output_token_ids detokenize exactly to the assistant text.
Stats
- 1,182,772 rows in 294 zstd parquet shards (~1.2 GB)
- 68.2% terminated with EOS (
status: success); 31.8% hit the 512-token cap (status: max_tokens) - 414M response tokens, mean 350 tokens/response
- Schema:
id(int64),conversations(user + assistant turns),output_token_ids(list<int32>),status
Decontamination
The corpus was scanned for benchmark contamination with word-level 10-gram shingle matching plus exact substring matching for short items, over both prompts and responses, with per-item coverage verification and manual review of hits.
Removed (14,639 rows, blocklist in `contaminated_row_ids.json`): all rows matching GSM8K test, MATH-500, HumanEval, or MT-Bench items. Re-scans of GSM8K test and HumanEval after removal return zero hits.
Known remaining contamination — do NOT evaluate on these: a broad 26-benchmark audit found verbatim test/validation items still present for the benchmarks below (blocklist shipped as eval_leak_row_ids.json, 6,647 rows, if you want to remove them yourself):
¹ mostly MetaMath-style number perturbations of the source problems rather than verbatim copies.
Most of the MMLU/MMLU-Pro/MATH overlap shares one provenance: Hendrycks MATH problems that exist in both the MMLU test set and MetaMath/MathInstruct-style augmentation data in the source blend.
Verified clean (zero hits at ≥0.5 coverage): GSM8K test, HumanEval, AIME 2024, AIME 2025, MGSM, LiveCodeBench, BigCodeBench, OpenBookQA, C-Eval, CMMLU.
Blind spots: GPQA main/extended are gated and were not scanned (GPQA-Diamond was scanned via a mirror: 1 verbatim + 3 partial hits). MMLU-Redux, AGIEval, Arena-Hard, and SWE-bench were not scanned. GSM8K train is near-fully present via MetaMath-style augmentation — expected for an SFT blend and not treated as contamination.
Intended use
Training draft/MTP heads for speculative decoding of Qwen3-4B and measuring acceptance length on clean benchmarks (GSM8K, AIME, code generation, or the held-out tail shards). If you evaluate model quality on MMLU/MATH-family or the other listed benchmarks, remove eval_leak_row_ids.json rows first.
Provenance & license
Prompts derive from the public open-perfectblend mixture (FLAN, evol/WizardLM-style, MetaMath/MathInstruct-style, UltraChat-style sources); responses are Qwen3-4B outputs. Use is subject to the licenses/terms of the upstream prompt sources and the Qwen3 model license (Apache-2.0).