Krsnapriya/CodeReviewEnv-Elite

🐞 DebugOps-RX: A Benchmark for Evaluating LLM Agents in Real-World Debugging Uncertainty

DebugOps-RX (Realistic eXecution) is an OpenEnv-compliant reinforcement learning benchmark. Current LLM SWE benchmarks overwhelmingly test for correctness given full context. DebugOps-RX tests decision-making under uncertainty, time pressure, and incomplete information.

This benchmark simulates the real-world Software Engineering / Site Reliability Engineering experience: an incident has occurred, logs are noisy or misleading, and the agent must explore a multi-file dependency network to find the truth, fix the bug, and verify it without wasting time.

1. Formal Task Distribution ($\mathcal{D}$)

A task $T$ is cleanly sampled from a distribution $\mathcal{D}$ over debugging environments:

$T \sim \mathcal{D}(n, b, \eta, \delta)$

where:

• —$n \in \mathbb{N}$: number of files in the repo ($2 \leq n \leq 20$)
• —$b \in \mathbb{B}$: bug type (from \{logic_error, key_error, dependency_error, state_corruption, stochastic_bug\})
• —$\eta \in [0,1]$: observability noise level
• —$\delta \in [0,1]$: temporal instability (environment drift)

By structuring dynamic procedural tasks around standard deviations of ($\eta$) and ($\delta$), you completely eradicate the opportunity for overfitted agent models.

2. Action & Observation Space

Action Space (Action)

Agents dispatch actions targeting the environment temporally sequentially:

• —open_file: Extracts the content of target into the observation window.
• —edit_file: Replaces or inserts content into the target file.
• —run_tests: Executes the integration suite yielding a pass/fail output.
• —analyze_logs: Fetches the application trace/logs (governed by $P(\eta)$).

Partial Observability (ObservableState)

The agent never sees the full truth. At any timestep t, the agent observes:

• —visible_files: A subset list of previously opened files.
• —logs: Trace outputs, subject to probabilistic corruption probabilities.
• —test_results: The diagnostic validation output.
• —time_remaining: Integer countdown before action threshold failure.

3. Formal Noise and Drift Models

Stop trying to handle "randomness" implicitly. DebugOps-RX mathematically dictates uncertainty:

Noise Model: Given noise level $\eta$: -$P(\text{log corruption}) = \eta$ -$P(\text{misleading attribution}) = 0.7\eta$ -$P(\text{extra irrelevant logs}) = \eta$

Temporal Drift Model: At each timestep $t$: -$P(\text{state change}) = \delta$

(State changes inject runtime flutter outputs into the log buffers making prior observations unstable).

4. Multi-Dimensional Vector Grading

Evaluating debugging capability requires a dense delayed reward system mapped to a multi-dimensional scalar Grade. We abandon purely binary success scoring.

• —Correctness (40%): Did the agent resolve the bug before step limits?
• —Efficiency (20%): Fraction of time budget remaining.
• —Robustness (20%): Inverse correlation against trashing wrong files sequentially.
• —Reasoning Quality (20%): Derived via structured mapping scalar $R$:

$$ R = \max(0,\ 0.5 \cdot \text{Exploration} + 0.3 \cdot \text{TestUsage} - 0.2 \cdot \text{Redundancy}) $$

Ultimately yielding the trajectory score scalar $S \in [0.0, 1.0]$.

5. Evaluation Protocol & Split Generalization

The true measure of a debugger agent is generalizing out of explicit memory constraints. DebugOps partitions evaluation bounds:

• —Train/Test Split: Standard logic_error and key_error bounds over standard code.
• —OOD Split: Generalizing over entirely new structural classifications (state_corruption, stochastic_bug).

🏎️ Running Locally

# Clone and install dependencies
pip install -r requirements.txt

# Run the statically multi-seeded benchmark script against the Random & Heuristic agents
python run_baseline.py

📊 Benchmark Results (Mean $\pm$ Std per 5 seeds)

[TRAIN SPLIT]
Agent        Easy         Medium       Hard         Extreme
----------------------------------------------------------------
Random     0.07±0.06    0.04±0.06    0.02±0.03    0.00±0.01    
Heuristic  0.77±0.01    0.18±0.01    0.23±0.00    0.79±0.00    

[OOD SPLIT]
Agent        Easy         Medium       Hard         Extreme
----------------------------------------------------------------
Random     0.07±0.05    0.03±0.06    0.02±0.03    0.00±0.01    
Heuristic  0.78±0.01    0.18±0.01    0.23±0.00    0.79±0.00