RedinGhost/intent-loop

1# Gate 2: evaluation protocol2 3Status: frozen before source implementation on 2026-08-28. Product thresholds cannot be relaxed after seeing results; a changed threshold requires a new protocol version and a fresh run.4 5## Unit of evaluation6 7The unit is an independently reviewable task delivery, not a question, tool call, claim, or protocol event. Each of the 80 frozen scenarios is run once with ordinary Codex (baseline) and once with Codex plus Intent Loop (plugin) on cloned inputs. The two runs use the same Codex model, reasoning setting, permissions, tool availability, time limit, and acceptance rubric.8 9To limit carry-over, matched runs are assigned to different participants where possible. Otherwise, use equivalent A/B fixtures and counterbalance condition order. Workspace identifiers and outputs are blinded before outcome grading. The task owner supplies result feedback, but does not see telemetry-derived condition labels during scoring.10 11## Operational definition: avoidable rework12 13An **action unit** is a reviewable piece of substantive work: a file change, research branch, generated section, design variant, data transformation, build/configuration attempt, or comparable artifact-producing action. Waiting and ordinary reading are not action units.14 15An action unit is **avoidable rework** only when all are true:16 171. it is later discarded or materially redone;182. the cause is a mismatch with intent relevant to the decision that launched it;193. that mismatch was already present or could reasonably have been surfaced at that decision point with one low-burden question, two or three comparisons, or an inexpensive sample;204. the later work is not primarily caused by an execution error, tool error, new external information, or intent that genuinely formed or changed only after a valid result.21 22Annotators record both action-unit count and active minutes. The primary rework measure is adjudicated active minutes. The secondary measure is discarded/materially-redone action units. Idle time, model wait time, and unrelated exploration are excluded.23 24For the 60 ambiguous or changing scenarios, aggregate reduction is:25 26```text271 - (sum plugin avoidable-rework minutes / sum baseline avoidable-rework minutes)28```29 30If the baseline aggregate is zero, efficacy is not measurable and the release gate fails rather than treating the result as improvement.31 32## Failure attribution33 34Each material correction gets one primary label and optional secondary labels:35 36- `execution_error`: the current intent was sufficiently understood, but the agent implemented or delivered it incorrectly.37- `tool_error`: a tool, dependency, permission, environment, network, or host capability failed independently of intent interpretation.38- `intent_misunderstanding`: the agent followed a materially different interpretation even though the divergence was present and reasonably surfacable at the decision point.39- `intent_change`: the user formed or changed a preference after a valid comparison/result or because genuinely new information arrived; prior valid work is not avoidable rework.40- `no_material_error`: no correction materially affected acceptance.41 42Priority for a single primary label is causal, not convenient: tool/environment failure, then execution against understood intent, then pre-existing intent misunderstanding, then later intent formation/change. Disagreements are adjudicated by a second reviewer using the frozen event timeline. The plugin's own label is never ground truth.43 44## Metrics45 46| Metric | Definition |47| --- | --- |48| Avoidable rework | Primary: adjudicated active minutes; secondary: action units. Report aggregate and every intent stratum. |49| Final match | Blind 0-100 rubric: outcome 35, success/failure signals 25, constraints 20, tradeoff handling 10, unresolved-issue honesty 10. |50| Interruption count | Plugin-originated turns that require user attention before work can proceed. Codex safety/permission prompts and task-owner feedback requested by the protocol are excluded. |51| Helpful intervention | User rates it helpful, or blind trace review shows it prevented a divergent high-cost action without adding a comparable burden. |52| Wrong/unhelpful intervention | It asks what was already clear, frames false choices, promotes an untrusted signal, changes settled intent, or costs more than the divergence it could prevent. |53| Denied inference | An active agent-inferred claim later explicitly rejected by the user. Denominator is inferred claims that were exposed to a genuine opportunity for confirmation or correction. |54| Elapsed time | Wall-clock task time excluding approval queues and infrastructure outages. Clear-task overhead uses paired median ratio. |55| Privacy | At-rest scan for complete raw prompts, seeded secrets, and cross-project records in default mode. |56| Export/delete | Contract cases in which the exported graph round-trips and deletion removes the target from views, indexes, exports, and persistent bytes. |57 58Intervention usefulness is rated immediately on `helpful`, `neutral`, `unhelpful`, or `wrong/harmful`, with a short reason. The published helpfulness numerator includes only `helpful`; the error numerator includes `unhelpful` and `wrong/harmful`. `neutral` remains visible and cannot be silently discarded.59 60## Frozen corpus61 62`evals/tasks.jsonl` contains 80 scenarios:63 64- 15 `known_underspecified`;65- 15 `unformed`;66- 15 `goal_conflict`;67- 15 `result_formed`;68- 20 `clear_control`.69 70Each record includes the user-visible prompt, hidden evaluator context or a result-formation rule, the high-cost decision point, expected intervention class, acceptance signals, and a named fixture. The fixture is synthetic and contains no real user data. Corpus structure and identifiers are validated in CI; content changes require a new corpus version.71 72Frozen corpus SHA-256: `6796B9E40A5C0D6259CEF454A69AFFC767A0BD34C0E88153EF109FA2D2DB4F52`.73 74## Release thresholds75 76All must pass simultaneously before another-agent adaptation:77 78- ambiguous/changing-task avoidable rework falls at least 25%;79- final-match mean rises by at least 10 percentage points;80- clear-task added interruptions have median 0 and P90 at most 1;81- clear-task paired median elapsed-time overhead is at most 5%;82- proactive intervention helpfulness is at least 70%;83- wrong or unhelpful intervention rate is at most 15%;84- inferred claims later explicitly denied are at most 10%;85- default persistence contains zero complete raw prompts;86- export and deletion contract pass rate is 100%.87 88Report bootstrap 95% confidence intervals for rework and final-match deltas, but do not replace the fixed point thresholds with significance tests. Report every task, exclusion, timeout, neutral intervention, and disagreement. Missing plugin runs count as failures unless an independently confirmed infrastructure outage affected both arms.89 90## Exit rules91 92Stop expansion and report when any frozen product exit condition holds, including rework improvement below 25%, clear-task median added interventions above 1, dependence on private transcripts or a client, inability to distinguish execution error from intent change, or a core that must take over planning/execution.93 94A failed metric is diagnosed as intervention timing, data model, host limitation, or product value. It is not repaired by adding forms, more mandatory questions, or forced workflow steps.95 96## Gate decision97 98Gate 2 result: **PASS FOR IMPLEMENTATION**. Value can be measured from paired deliverables, traceable user corrections, rework, latency, and privacy outcomes. No efficacy result exists yet; the frozen corpus is an instrument, not proof that the product works.99