vedkdev/FlakyTestSleuthOpenEnvRL

1# FlakySleuth Grading: Exact Scoring Formulas2 3This document describes the **exact scoring logic implemented in code** for:4- Task 1: `classify` (`classify_flakiness`)5- Task 2: `root_cause` (`classify_root_cause`)6- Task 3: `fix_proposal` (`propose_fix`)7 8It also explains how per-step rewards are combined inside the environment.9 10## Source of Truth11 12- `env/environment.py`13- `graders/__init__.py`14- `graders/task1_grader.py`15- `graders/task2_grader.py`16- `graders/task3_grader.py`17- `dataset/category_similarity.json`18 19## 1) Dispatch: Which grader is used?20 21`graders/grade_action()` selects grader by `task["task_type"]`:22- `classify` -> Task 1 grader23- `root_cause` -> Task 2 grader24- `fix_proposal` -> Task 3 grader25- anything else -> `0.0`26 27## 2) Environment reward pipeline (applies to all tasks)28 29At each `env.step(action)`:30 311. If action is terminal (`classify_flakiness`, `classify_root_cause`, `propose_fix`):32   - compute `terminal_score = grade_action(action, task)`33   - compute penalties34   - final step reward:35 36```text37reward = clamp(38    cumulative_progress + terminal_score - late_penalty - wrong_dir_penalty,39    0.0,40    1.041)42```43 44Where:45- `late_penalty = max(0, step_count - 15) * 0.05`46- `wrong_dir_penalty = 0.2` only when:47  - action is `classify_flakiness`48  - predicted argument is `"stable"`49  - ground-truth label is `"flaky"`50- `done = True`51 522. If action is non-terminal (exploration):53   - compute `progress` from exploration action54   - update cumulative progress:55 56```text57cumulative_progress = clamp(cumulative_progress + progress, 0.0, 0.30)58reward = progress59```60 613. Timeout rule:62   - if not already done and `step_count >= max_steps`, set `done = True`63   - no additional terminal score is applied at timeout.64 65## 3) Exploration progress rewards (exact values)66 67### `read_file`68- file missing/unsafe -> `progress = -0.05`69- file already read in this episode -> `progress = 0.0`70- new file:71  - if file path contains `task["test_file"]` -> `0.07`72  - else if file ends with `.py` -> `0.03`73  - else -> `0.01`74 75### `search_code`76- base reward:77  - if query contains any flaky-signal tokens (`sleep`, `random`, `time`, `datetime`, `thread`, `asyncio`, `fixture`, `setup`, `teardown`, `global`, `shared`, `singleton`, `os.environ`, `socket`, `timeout`, `retry`, `mock`, `patch`) -> `0.04`78  - otherwise -> `0.01`79- spam penalties (all apply, then summed and capped):80  - repeated same normalized search pattern in episode:81    - `repeat_penalty = min(0.02 * (pattern_count - 1), 0.12)` for `pattern_count > 1`82  - repeated same search context (same normalized pattern + same extracted top `.py` hit files):83    - `context_penalty = min(0.03 * (context_count - 1), 0.15)` for `context_count > 1`84  - long search-only streak:85    - `streak_penalty = min(0.02 * (consecutive_searches - 3), 0.20)` for `consecutive_searches > 3`86  - total spam penalty cap: `min(sum_penalties, 0.35)`87- final `search_code` progress:88 89```text90progress = max(-0.25, base_reward - spam_penalty)91```92 93- environment appends `WARNING:` text to tool output when penalties fire.94- `consecutive_searches` resets on any non-`search_code` action.95 96### `run_test`97- if category is **not** one of `OD`, `OD-Brit`, `OD-Vic` -> `0.05`98- if category is order-dependent (`OD`, `OD-Brit`, `OD-Vic`) -> `0.0`99 100### unsupported action type101- `progress = -0.05`102 103## 4) Task 1 scorer (`classify_flakiness`)104 105Binary exact-match scorer:106 107```text108if action_type != "classify_flakiness": return 0.001109if predicted not in {"flaky","stable"}: return 0.001110truth = task["label"] (default "flaky")111terminal_score = 0.999 if predicted == truth else 0.001112```113 114Notes:115- In current dataset builder, rows are written with `label = "flaky"` by default.116- Predicting `"stable"` on flaky truth also triggers environment `wrong_dir_penalty = 0.2`.117 118## 5) Task 2 scorer (`classify_root_cause`)119 120Matrix-based similarity scorer.121 122### 5.1 Category normalization123 124Prediction and truth are normalized by:125- trim126- replace `_` with `-`127- replace spaces with `-`128- uppercase and map through canonical aliases:129  - `OD-BRIT` -> `OD-Brit`130  - `OD-VIC` -> `OD-Vic`131  - etc.132 133If normalized value is not in valid set, score is `0.001`.134 135Truth category is the **first** category if semicolon-separated:136 137```text138raw_truth = str(task["category"]).split(";")[0]139```140 141### 5.2 Similarity scoring142 143```text144if predicted == truth: return 0.999145else return clamp(similarity[predicted,truth] or similarity[truth,predicted] or 0.0, 0.001, 0.999)146```147 148The similarity matrix is loaded from `dataset/category_similarity.json`.149 150Current non-identity similarity entries:151- `OD,OD-Brit`: `0.7`152- `OD,OD-Vic`: `0.7`153- `OD-Brit,OD-Vic`: `0.8`154- `OD,NIO`: `0.4`155- `OD,NDOI`: `0.3`156- `NOD,TD`: `0.6`157- `NOD,TZD`: `0.5`158- `NOD,NDOI`: `0.5`159- `TD,TZD`: `0.7`160- `NOD,ID`: `0.3`161- `UD,OD`: `0.2`162- `UD,NOD`: `0.2`163- `UD,NIO`: `0.2`164- `UD,TD`: `0.2`165- `UD,ID`: `0.2`166 167Any missing pair defaults to `0.0`.168 169## 6) Task 3 scorer (`propose_fix`)170 171Hybrid weighted scorer:172 173```text174if action_type != "propose_fix": return 0.001175if proposed_fix is empty: return 0.001176 177total = 0.35 * pattern_score + 0.25 * apply_score + 0.40 * judge_score178terminal_score = round(clamp(total, 0.001, 0.999), 4)179```180 181### 6.1 `pattern_score`182 183Category-specific keyword patterns are checked against the proposed diff.184 185For category with pattern list:186 187```text188matches = number of patterns found (case-insensitive substring)189pattern_score = min(0.999, matches / max(1, len(patterns) * 0.4))190```191 192If category has no pattern list:193- `pattern_score = 0.5`194 195Current pattern lists:196- `TD`: `freeze_time`, `mock`, `patch`, `utcnow`, `datetime`, `monkeypatch`197- `TZD`: `timezone`, `utc`, `pytz`, `zoneinfo`, `tzinfo`, `UTC`198- `NOD`: `seed`, `mock`, `patch`, `deterministic`, `sorted`199- `NIO`: `setup`, `teardown`, `fixture`, `yield`, `cleanup`, `autouse`200- `ID`: `sorted(`, `list(`, `frozenset`, `OrderedDict`201 202### 6.2 `apply_score` (`_check_diff_applies`)203 204```text205if diff does not contain both '---' and '+++': return 0.001206if sandbox_root missing or not existing: return 0.3207else run: patch --dry-run -p1 -i <temp_patch>208  return 0.999 if patch exit code == 0209  return 0.001 otherwise210on exception: return 0.3211```212 213### 6.3 `judge_score` (`_llm_judge`)214 215LLM judge behavior:216- If no API key available -> `judge_score = 0.5`217- Else sends a judge prompt asking for JSON `{"score": 0..10, "reason": ...}`218- Parses integer score, clamps to `[0,10]`, then scales to `[0,1]`:219 220```text221judge_score = clamp(int_score, 0, 10) / 10222```223 224- On any judge exception / parse failure -> `judge_score = 0.5`225 226API/model resolution in judge:227- API key preference: `API_KEY` -> `OPENROUTER_API_KEY` -> `OPENAI_API_KEY`228- Base URL:229  - OpenRouter inferred -> `https://openrouter.ai/api/v1`230  - else -> `https://api.openai.com/v1`231- Model default:232  - OpenRouter base URL -> `qwen/qwen3.6-plus:free`233  - else -> `gpt-4o-mini`234 235## 7) Worked examples236 237### Example A: Task 1 correct classify early238 239- `cumulative_progress = 0.05`240- `terminal_score = 0.999`241- `late_penalty = 0.0`242- `wrong_dir_penalty = 0.0`243 244```text245reward = clamp(0.05 + 0.999 - 0 - 0, 0, 1) = 0.999246```247 248### Example B: Task 2 wrong category but some exploration249 250- `cumulative_progress = 0.05`251- `terminal_score = 0.001` (no similarity match)252- penalties = `0`253 254```text255reward = clamp(0.05 + 0.001, 0, 1) = 0.051256```257 258### Example C: Task 3 with weak fix and no API key259 260- `judge_score = 0.5` fallback261- `apply_score` and `pattern_score` depend on diff contents262- final weighted sum then clamped and rounded to 4 decimals.263 264## 8) Important implementation notes265 266- `cumulative_progress` is capped at `0.30` and never below `0.0`.267- Terminal reward can be reduced by late penalty after step 15.268- Timeout does not invoke grader; it only ends the episode.269- Dataset construction choices (especially `label` and category quality) heavily influence observed score behavior.270 271## 9) Inference-side controls (not grader formulas)272 273`inference.py` now includes policy/runtime controls that do not change grader math directly but change agent behavior:274 275- episode memory injected into every prompt (recent files, search patterns, no-progress streak)276- explicit loop warning prompt when no-progress/duplicate patterns are detected277- duplicate `read_file` attempts are overridden to targeted `search_code`278- conversation compaction controls:279  - `--history-prune-start-step` (default `12`)280  - `--history-window-turns` (default `4`)281  - `--history-max-chars` (default `50000`)282- detailed tracing options (`--trace-agent`, `--trace-prompts`) for audit/debug283