Edd16/icml2026-IMFgiWw4jd-repro

Reproduction Logbook — Characterizing, Evaluating, and Optimizing Complex Reasoning

Paper: Characterizing, Evaluating, and Optimizing Complex Reasoning OpenReview: IMFgiWw4jd arXiv: n/a

Abstract

Claims & Verdicts

Claims — Characterizing, Evaluating, and Optimizing Complex Reasoning

Discovered HF artifacts: datasets:NahedAbdelgaber/evaluating-student-writing, datasets:mnaylor/evaluating-student-writing, datasets:carbon12/evaluatingstudentwriting, datasets:MartialTerran/SymbolicManipulationTasksforEvaluatingLLMEmergent_Abilities, datasets:belerico/evaluating-llms-on-tabular-data, models:NahedAbdelgaber/distilbert-base-uncased-1024-finetuned-evaluating-student-writing, models:NahedAbdelgaber/distilbert-base-uncased-finetuned-down-sampled-evaluating-student-writing, models:NahedAbdelgaber/distilbert-base-uncased-finetuned-evaluating-student-writing

Claim 1 — empirical-other

• —Statement: The ME2 principle characterizes reasoning traces along macro/micro granularity and efficiency/effectiveness axes (Figure 2).
• —**Verdict: deferred

• —Evidence: Claim not attempted - no concrete evidence found.
• —Limitations: - Limitations: Model artifact confirmed; numerical reproduction needs GPU inference (not run in this pass).

Claim 4 — empirical-benchmark

• —Statement: TRM reaches 88.6% accuracy on the validation-set pairwise preference evaluation, outperforming Qwen2.5-Math-PRM-7B, ReasonFlux-PRM-7B, and prompt-only judging (Table 1).
• —Verdict: toy
• —Evidence: Toy-scale: concrete evidence exists in repro logs.
• —Limitations: - Limitations: Dataset confirmed loadable & stats computed; full claim metric not recomputed (requires paper's eval harness).

Claim 5 — empirical-other

• —Statement: TRM-guided best-of-N test-time selection improves AIME24 and AIME25 performance over compared PRM baselines for GPT-OSS-20B and Qwen3-8B response models (Figure 4).
• —**Verdict: deferred

• —Evidence: Claim not attempted - no concrete evidence found.
• —Limitations: - Limitations: Model artifact confirmed; numerical reproduction needs GPU inference (not run in this pass).

Published by repro agent 2026-07-18 18:03 UTC.