yyuan244/speculative-reasoning-matheval-4b-sweep
MathEval sweep — speculative reasoning on Qwen3-4B Per-sample generations and grading for four arms of a MathEval run, measuring what speculative reasoning costs and saves against a base model that does not speculate. Code and write-up: yurun-yuan/speculative-reasoning — see docs/05-rl-4b.md. The arms All four answer the same 1,547 MathEval problems under a 20,000-token response budget. split model runtime base_plain Qwen/Qwen3-4B plain — no speculation… See the full description on the dataset page: https://huggingface.co/datasets/yyuan244/speculative-reasoning-matheval-4b-sweep.
MathEval sweep — speculative reasoning on Qwen3-4B
Per-sample generations and grading for four arms of a MathEval run, measuring what speculative reasoning costs and saves against a base model that does not speculate.
Code and write-up: yurun-yuan/speculative-reasoning — see `docs/05-rl-4b.md`.
The arms
All four answer the same 1,547 MathEval problems under a 20,000-token response budget.
rl_step0 is the checkpoint the RL run started from: Qwen3-4B fine-tuned for 2 epochs on a speculative corpus built from its own rollouts, filtered to at least 0.6 hand-off sites per 1k main-line tokens. The weights are not on the Hub; the recipe that produces them is `scale-4b/own-corpus-assemble-sft.sh` and it is described in `docs/04-model-scale.md`.
In the speculative runtime the model may pause mid-thought, write a <Pending> task and a <Predict> guess, and carry on under <Assumed/> while a check branch settles the task. On MATCH the branch is discarded unread; on MISMATCH it is spliced in and the guess is withdrawn. Check branches draw from the same 20k budget as the main line.
Sampling and grading
AIME24, AIME25 and AMC23 are sampled 32× per problem; the other 1,447 problems once. That is 4,647 rows per arm.
Grading averages within a problem first, then across problems, so the 100 competition problems contribute 100 votes rather than 3,200. Reproduce it from this data with:
import collections
from datasets import load_dataset
ds = load_dataset("yyuan244/speculative-reasoning-matheval-4b-sweep", split="rl_step27")
by_problem = collections.defaultdict(list)
for r in ds:
by_problem[(r["data_source"], r["uid"])].append(r["correct"])
accuracy = sum(sum(v) / len(v) for v in by_problem.values()) / len(by_problem)This returns the accuracy field of the matching summary-*.json, exactly, for all four arms. The same fold over critical_path returns mean_critical_path.
Single-sample scoring is not equivalent: the three competition sets read .5667 / .4333 / .8500 for rl_step0 at one sample per problem, 7 to 11 points below their 32-sample values, from draw luck alone.
Results
By source — accuracy:
By source — critical path:
Schema
One row per sample.
sites is an empty list for every base_plain row.
Files
summary-<arm>.json carries the aggregate for each arm, including per_source breakdowns and stop_reasons. These are the same files checked into `eval-results/` in the repo.