chichi56/deepmath-l5-9-qwen3-4b-at8-profile

DeepMath-L5-9 10K @8 Profiled by Qwen3-4B-Instruct-2507

Difficulty-stratified rollout profile of 10,000 DeepMath problems (levels 5-9) by Qwen3-4B-Instruct-2507, 8 rollouts per question (n=8, T=0.7, top_p=0.95, max_tokens=16384).

Built for the OPSD context-strength study: comparing two OPSD context sources (gold ref-solution vs hint sequence) across three difficulty buckets. The core hypothesis: when the prepended context is too strong, OPSD degrades into SFT — the student is just copying the context tokens rather than internalizing reasoning.

Bucket definition

Full pass_k distribution: 0:756 · 1:276 · 2:244 · 3:260 · 4:288 · 5:347 · 6:523 · 7:995 · 8:6311.

Files

• —rollouts_10k.parquet (207 MB) — full per-question rollouts
• —id prompt (json messages) gold completions (list[str], len=8) graded (list[int 0/1], len=8) pass_k (int 0-8) bucket (str)
• —buckets.jsonl (688 KB) — compact id → pass_k → bucket
• —bucket_stats.json — counts and means

Generation config

model        : Qwen3-4B-Instruct-2507
data source  : DeepMath-103K, levels 5-9, deepmath_L5to9_10k
n            : 8 rollouts per question
temperature  : 0.7
top_p        : 0.95
max_tokens   : 16384
max_model_len: 16384
seed         : 1234
hardware     : 8 × AMD MI300X, vllm DP=8 TP=1
grader       : matharena + Qwen2.5-Math grader merged
              (experiment/eval/math_grader.py from OPD-Done-Right)

Use

from datasets import load_dataset
ds = load_dataset("chichi56/deepmath-l5-9-qwen3-4b-at8-profile", split="train")
print(ds[0]["bucket"], ds[0]["pass_k"])

License

MIT. DeepMath itself is released by its authors; we redistribute only question ids, model rollouts, and our grading.