pantomiman/reason-over-search-eval-m5

Reason-over-Search M5: unified held-out evaluation (9 runs)

Held-out 7-benchmark QA evaluation of the M5 reward-shape x seed ablation: Qwen3.5-0.8B GRPO-trained on MuSiQue with three reward shapes (F1-only / F1+format / EM-only) at three seeds (42 / 43 / 44). This repo is the uniform, light index of every checkpoint's eval scores; it carries the per-cell metric_score.txt + config.yaml and the per-run aggregated CSVs, in ONE consistent layout. (The heavy intermediate_data.json per-question generations stay in the original repo, pantomiman/reason-over-search-eval-results, which is unchanged.)

The 9 runs

Total: 222 checkpoints, every one evaluated on all 7 benchmarks = 1,554 (checkpoint, dataset) eval cells, 0 missing. Best avg EM is over the 7-dataset mean, length-uncapped (each run's own best checkpoint). Benchmarks: NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle (4 test + 3 dev splits, 51,713 rows per checkpoint).

Layout (uniform across all 9 runs)

<run>/step_<N>/<dataset>/metric_score.txt   # em / acc / f1
<run>/step_<N>/<dataset>/config.yaml         # exact eval config for that cell
csv/<run>.csv                                # per-step aggregate: per-dataset em/f1 + avg_em/avg_f1

<run> is one of the 9 above (m5_<1|5|6>_s<42|43|44>); <N> steps every 10. This unifies the two legacy layouts in reason-over-search-eval-results (seed-42 used timestamped run-dirs m9_2/<dataset>/<ts>_step100_seed1/; seed-43/44 used m9_5/<run>/step_N/<dataset>/).

Key result

At matched horizons (h=180, h=230), EM-only is the worst reward shape at all three seeds; both F1 variants beat it by ~7-8 pp avg EM. F1+format is the most seed-robust (h=230 spread 1.4 pp across seeds vs 4.9 pp for F1-only). The reward-shape ranking between the two F1 variants inverts by seed (F1+format > F1-only at seeds 42/43; F1-only > F1+format at seed 44). The single best checkpoint is F1-only seed-44 (0.352 @step310), but that lead is largely a training-budget effect (only +0.5 pp at matched h=180). Full length-normalised analysis: see the project repo `docs/report/RESULTSm95deep.md`.

Provenance + reproducibility notes

• —Eval pipeline: FlashRAG fork, byte-aligned prompt qwen35_m5_train, greedy decode, E5-base-v2 + Wiki-18 IVF-SQ8 retriever.
• —Backends: seed-42 + seed-43/44 (m95a..e) served via SGLang or the SGLang->vLLM proxy on the rift box; the seed-43 EM-only leg (m56_s43) was served SGLang-native + triton attention on ALICE A100 MIG (a minor cross-backend confound, greedy decode makes it near-equivalent).
• —Source training checkpoints + the per-question intermediate_data.json: see the Collection linked from the training-index repo.