cy0307/awesome-loop-engineering

1# Benchmark Optimization Loop2 3## Objective4 5Improve a measurable system outcome through small experiments while preserving correctness and keeping every accepted change reproducible.6 7## Use This When8 9- You have a stable benchmark, eval suite, or objective metric.10- One change can be tested independently of the next.11- The system can reject regressions and restore the last accepted state.12 13Do not use this loop when the score is easy to game, the benchmark is still changing, or the real outcome requires human judgment that the metric does not capture.14 15## Trigger16 17- Schedule: a bounded overnight or weekly experiment window.18- Event: a new baseline, model, dataset, or optimization target is available.19- Manual bootstrap: "run up to five benchmark-backed optimization experiments."20 21## Intake22 23- Baseline artifact, benchmark command, correctness checks, and target metric.24- Prior experiment ledger, failed hypotheses, and protected files.25- Compute, token, time, and concurrency budgets.26 27## Agents28 29- Experiment designer: proposes one falsifiable change and expected effect.30- Implementer: applies only that change in an isolated candidate workspace.31- Verifier: runs correctness checks and the frozen benchmark.32- Recorder: accepts or rejects the candidate and updates the experiment ledger.33 34## Workspace And Permissions35 36- Use a disposable worktree, branch, or sandbox per candidate.37- Allow edits only to the declared optimization surface.38- Keep benchmark data, scoring code, holdout cases, and acceptance thresholds read-only.39- Disallow test deletion, scorer edits, hidden-test inspection, and concurrent candidates that exceed the budget.40 41## Durable State42 43- Baseline version and score, hypothesis, candidate diff, commands, raw results, cost, decision, and rejection reason.44- The last accepted artifact remains the parent of the next experiment.45 46## Loop Steps47 481. Freeze the baseline, benchmark version, correctness gate, and budget.491. Read the ledger so the next hypothesis does not repeat a failed experiment.501. Propose one bounded change with a predicted effect.511. Apply it in an isolated candidate workspace.521. Run correctness checks before the benchmark.531. Compare repeated benchmark runs against the accepted baseline.541. Accept only a reproducible improvement that clears the minimum delta; otherwise reject and restore the baseline.551. Record the full receipt and continue until the target or budget is reached.56 57## Verification Gates58 59- Correctness and safety checks pass unchanged.60- The benchmark, dataset split, scorer, and environment match the recorded baseline.61- Improvement clears the declared minimum delta across the required repeats.62- The candidate does not worsen protected secondary metrics beyond tolerance.63- The ledger contains enough evidence to reproduce the decision.64 65## Budget And Exit66 67- Max retries: 5 candidate experiments.68- Max runtime: 240 minutes.69- Stop on target attainment, budget exhaustion, two repeated hypotheses, benchmark instability, or a protected-metric regression.70 71## Escalation72 73Escalate when the metric conflicts with observed quality, a candidate changes the evaluator, results vary beyond tolerance, or the next experiment would broaden permissions or compute.74 75## Loop Instruction76 77```text78Optimize <artifact> against <benchmark> for at most five experiments.79Treat <correctness command> as a non-negotiable gate and <metric> as the optimization signal.80Change one declared variable per candidate in an isolated workspace. Never edit the81benchmark, scorer, holdout set, or protected tests. Accept a candidate only when repeated82runs improve the baseline by <minimum delta> without regressing <secondary metrics>.83Record every hypothesis, diff, command, score, cost, and accept/reject decision.84```85 86## Worked Example87 88A team wants to reduce an agent workflow's median latency without lowering task success. The loop starts from a frozen 78% success / 42-second baseline, tries one routing or caching change at a time, reruns the same 100-task suite three times, and accepts only candidates that keep success within one percentage point while cutting median latency by at least 5%.89 90## Failure Modes91 92- Optimizing on the same cases used to invent the change.93- Accepting a noisy single run as improvement.94- Changing multiple variables and losing causal attribution.95- Improving the headline metric while silently degrading cost, safety, or tail latency.96- Letting the agent modify the verifier that judges its own work.97 98## Example Contract99 100- [`examples/benchmark-optimization-loop.json`](../examples/benchmark-optimization-loop.json)101 102## References103 104- [A Self-Improving Coding Agent](https://arxiv.org/abs/2504.15228) - Demonstrates benchmark-gated self-modification with measurable gains.105- [Understanding the Challenges in Iterative Generative Optimization with LLMs](https://arxiv.org/abs/2603.23994) - Identifies evaluation and credit-assignment choices that make iterative optimization brittle.106