SGK86/procedural-graphs-repro

Procedural Graphs — Reproduction (HotpotQA, Qwen2.5-72B-Instruct)

Scaled-down reproduction of "Procedural Graphs: Self-Evolving Execution Structures for LLM Agents" (arXiv 2609.09153, Google, 2026) on HotpotQA (distractor), training-free: a directed Procedural Graph with condition/guidance/pitfalls edges, locate→extract→generate guidance at inference, and offline self-evolution with a validation gate.

Run: 2026-09-21, solver Qwen/Qwen2.5-72B-Instruct via HF Inference. Pools: 96 train / 48 val / 96 test HotpotQA validation items (streaming, seed 13).

Results

Δ vs. baseline: +4.17 EM, +3.11 F1 after 3 rounds.

Evolution trace (gate on held-out F1):

• —Round 1: REJECT — val F1 67.59 → 66.04 (edits: addedge collectfacts→composeanswer, editedge crosscheck→composeanswer); rolled back, added to rejection memory
• —Round 2: REJECT — val F1 67.59 → 66.14 (editedge crosscheck→compose_answer); rolled back
• —Round 3: ACCEPT — val F1 67.59 → 68.92 (editedge crosscheck→compose_answer refined by two prior rejections)

Notes:

• —The paper reports HotpotQA 87.3 EM with a frontier solver (Gemini 3.1 Pro) and its own benchmark harness; this reproduction differs in scale (96-item pools, open 72B model, ~9k-char context cap), so the absolute numbers are not comparable — the claim reproduced is the relative improvement from self-evolution and the behavior of the validation gate (2 of 3 mutations rejected).
• —4 guidance JSON parse failures occurred mid-run; the agent's designed fallback (run guidance-less) kept the run alive.

Files

• —graph.py — ProceduralGraph, 2-hop subgraph extraction, refiner edit application, HotpotQA skeleton
• —agent.py — GuidedAgent (locate → 2-hop extract → generate guidance → solve, with fallback)
• —evolver.py — evolution round: diagnostic rollout, refiner mutation, validation gate, rejection memory
• —llm.py — HF Inference backend with model fallback + retry
• —hotpot.py — EM/F1 (standard normalization) + pool loading
• —run_pg.py — main script

Run it: HF_TOKEN=... python run_pg.py (deps: huggingface_hub, datasets).