vedkdev/FlakyTestSleuthOpenEnvRL
0
1# FlakySleuth — Comprehensive Round 1 Build Plan2## Meta × PyTorch × Scaler OpenEnv Hackathon3 4---5 6## 0. What You Are Building (One Paragraph for Clarity)7 8You are building an **OpenEnv-compliant RL environment** called `FlakySleuthEnv`. It simulates a real software engineering task: investigating flaky tests in real Python GitHub repositories. An LLM agent is dropped into a sandboxed repo at a specific commit, given a test that is known to be flaky (sourced from the IDoFT dataset), and must use tool calls (read files, grep code, run tests) to investigate and produce a verdict. The environment scores the agent's verdict using deterministic graders (Tasks 1 and 2) and a hybrid programmatic + LLM judge grader (Task 3). You are NOT training any model. The submitted artifact is the environment itself — its graders, reward logic, OpenEnv spec compliance, Docker container, and a baseline `inference.py` script that proves it works.9 10---11 12## 1. Repository Structure13 14```15flaky-sleuth-env/16│17├── inference.py ← REQUIRED: must be named exactly this, in root18├── openenv.yaml ← REQUIRED: OpenEnv spec metadata19├── Dockerfile ← REQUIRED: must build and run20├── requirements.txt21├── README.md22│23├── server.py ← FastAPI HTTP server (OpenEnv endpoints)24│25├── env/26│ ├── __init__.py27│ ├── models.py ← All Pydantic models (Observation, Action, Reward)28│ ├── environment.py ← FlakySleuthEnv core class29│ ├── sandbox.py ← Git clone, file read, grep, run_test30│ └── task_loader.py ← Loads tasks from dataset CSV31│32├── graders/33│ ├── __init__.py ← grade_action() dispatcher34│ ├── task1_grader.py ← Binary flaky/stable35│ ├── task2_grader.py ← Root cause category + similarity matrix36│ └── task3_grader.py ← Fix proposal: pattern + diff + LLM judge37│38├── dataset/39│ ├── build_dataset.py ← OFFLINE SCRIPT: preprocess IDoFT → py_tasks.csv40│ ├── py_tasks.csv ← Final preprocessed task bank (committed to repo)41│ └── category_similarity.json ← Similarity matrix for Task 2 partial credit42│43└── tests/44 └── test_compliance.py ← openenv validate compliance checks45```46 47---48 49## 2. Data Pipeline (Do This First, Offline)50 51### 2.1 Download the Raw Dataset52 53```bash54git clone https://github.com/TestingResearchIllinois/idoft55# The file you need:56# idoft/py-data.csv57```58 59### 2.2 Understand the CSV Columns60 61The `py-data.csv` has these columns:62```63Project URL | SHA Detected | Pytest Test Name | Category | Status | PR Link | Notes64```65 66- **Project URL**: GitHub repo to clone67- **SHA Detected**: Exact commit to clone at (this is where the test IS flaky)68- **Pytest Test Name**: Format is `path/to/test_file.py::TestClass::test_method` or `path/to/test_file.py::test_method`69- **Category**: One of OD, OD-Brit, OD-Vic, NIO, NOD, UD, TD, TZD, ID, NDOI, NDOD, OSD (may be semicolon-separated for multiple)70- **Status**: Blank, Opened, Accepted, Rejected, etc.71- **PR Link**: Format `owner/repo#number` — only present when Status is Opened/Accepted72 73### 2.3 Filter Rules Per Task74 75```python76# Task 1 (classify): Use these categories — they have clear static signals77TASK1_CATEGORIES = ["NOD", "TD", "TZD", "NIO", "ID", "OD", "OD-Brit", "OD-Vic"]78 79# Task 2 (root cause): Same categories — agent must identify which one80TASK2_CATEGORIES = ["NOD", "TD", "TZD", "NIO", "ID", "OD", "OD-Brit", "OD-Vic"]81# Exclude "UD" (unknown — no ground truth to grade against)82 83# Task 3 (fix proposal): ONLY rows where a fix was accepted AND category is gradeable84TASK3_CATEGORIES = ["TD", "TZD", "NOD", "NIO", "ID"]85# Exclude: OD, OD-Brit, OD-Vic (cannot verify fix without multi-order execution)86# Exclude: UD (unknown cause = cannot score fix)87# Require: Status == "Accepted" AND PR Link is not empty88```89 90### 2.4 Build `py_tasks.csv` (the `build_dataset.py` script)91 92This script runs ONCE offline. It:931. Reads `idoft/py-data.csv`942. For each row, fetches the test source code by cloning the repo at SHA (or using GitHub raw API)953. For Task 3 rows (Status=Accepted), fetches the PR diff from GitHub API964. Outputs `dataset/py_tasks.csv`97 98```python99# dataset/build_dataset.py100 101import pandas as pd102import requests103import subprocess104import tempfile105import os106 107GITHUB_TOKEN = os.environ["GITHUB_TOKEN"] # set this before running108 109def fetch_test_code(repo_url: str, sha: str, pytest_test_name: str) -> str:110 """111 Clone repo at SHA, extract the test function source code.112 pytest_test_name format: path/to/test.py::TestClass::test_method113 """114 test_file = pytest_test_name.split("::")[0]115 with tempfile.TemporaryDirectory() as tmpdir:116 subprocess.run([117 "git", "clone", "--depth=1", repo_url, tmpdir118 ], capture_output=True)119 subprocess.run([120 "git", "checkout", sha121 ], cwd=tmpdir, capture_output=True)122 filepath = os.path.join(tmpdir, test_file)123 if not os.path.exists(filepath):124 return ""125 with open(filepath) as f:126 return f.read()[:5000] # cap at 5000 chars127 128 129def fetch_pr_diff(pr_link: str) -> str:130 """131 pr_link format: "owner/repo#number"132 Returns unified diff string of the PR.133 """134 if not pr_link or "#" not in pr_link:135 return ""136 repo, number = pr_link.strip().split("#")137 url = f"https://api.github.com/repos/{repo}/pulls/{number}"138 headers = {139 "Authorization": f"token {GITHUB_TOKEN}",140 "Accept": "application/vnd.github.diff"141 }142 resp = requests.get(url, headers=headers, timeout=10)143 if resp.status_code == 200:144 return resp.text[:3000] # cap diff size145 return ""146 147 148def build():149 df = pd.read_csv("idoft/py-data.csv")150 151 # Rename columns for clarity152 df.columns = [c.strip() for c in df.columns]153 154 rows = []155 for _, row in df.iterrows():156 repo_url = str(row.get("Project URL", "")).strip()157 sha = str(row.get("SHA Detected", "")).strip()158 test_name = str(row.get("Pytest Test Name", "")).strip()159 category_raw = str(row.get("Category", "")).strip()160 status = str(row.get("Status", "")).strip()161 pr_link = str(row.get("PR Link", "")).strip()162 163 # Skip rows with missing essentials164 if not repo_url or not sha or not test_name or not category_raw:165 continue166 167 # Take primary category (first if semicolon-separated)168 category = category_raw.split(";")[0].strip()169 170 # Skip UD for Task 2 (no ground truth)171 if category == "UD":172 continue173 174 # Determine task types this row is eligible for175 task_types = []176 if category in ["NOD", "TD", "TZD", "NIO", "ID", "OD", "OD-Brit", "OD-Vic"]:177 task_types.append("classify")178 task_types.append("root_cause")179 if (category in ["TD", "TZD", "NOD", "NIO", "ID"]180 and status == "Accepted"181 and pr_link and pr_link != "nan"):182 task_types.append("fix_proposal")183 184 if not task_types:185 continue186 187 # Fetch test source code188 test_code = fetch_test_code(repo_url, sha, test_name)189 if not test_code:190 continue191 192 # Fetch fix diff for Task 3 eligible rows193 known_fix_diff = ""194 if "fix_proposal" in task_types:195 known_fix_diff = fetch_pr_diff(pr_link)196 197 rows.append({198 "repo_url": repo_url,199 "sha": sha,200 "test_name": test_name,201 "test_file": test_name.split("::")[0],202 "category": category,203 "status": status,204 "pr_link": pr_link,205 "task_types": ";".join(task_types),206 "test_code": test_code,207 "known_fix_diff": known_fix_diff,208 })209 210 out = pd.DataFrame(rows)211 out.to_csv("dataset/py_tasks.csv", index=False)212 print(f"Built {len(out)} task rows")213 print(out["category"].value_counts())214 print(out["task_types"].value_counts())215 216if __name__ == "__main__":217 build()218```219 220### 2.5 Build `category_similarity.json`221 222```json223{224 "OD,OD-Brit": 0.7,225 "OD,OD-Vic": 0.7,226 "OD-Brit,OD-Vic": 0.8,227 "OD,NIO": 0.4,228 "OD,NDOI": 0.3,229 "NOD,TD": 0.6,230 "NOD,TZD": 0.5,231 "NOD,NDOI": 0.5,232 "TD,TZD": 0.7,233 "NOD,ID": 0.3,234 "UD,OD": 0.2,235 "UD,NOD": 0.2,236 "UD,NIO": 0.2,237 "UD,TD": 0.2,238 "UD,ID": 0.2239}240```241 242---243 244## 3. Pydantic Models (`env/models.py`)245 246```python247from pydantic import BaseModel248from typing import Literal, Optional, List249 250class FlakySleuthObservation(BaseModel):251 repo_url: str252 test_name: str253 test_code: str254 file_tree: List[str]255 tool_output: Optional[str] = None256 task_type: Literal["classify", "root_cause", "fix_proposal"]257 task_description: str258 step_count: int259 260class FlakySleuthAction(BaseModel):261 action_type: Literal[262 "read_file",263 "search_code",264 "run_test",265 "classify_flakiness",266 "classify_root_cause",267 "propose_fix",268 ]269 argument: str270 271class FlakySleuthReward(BaseModel):272 score: float273 breakdown: dict274 explanation: str275```276 277---278 279## 4. Sandbox (`env/sandbox.py`)280 281The sandbox wraps a cloned git repo. It handles all filesystem operations.282 283```python284import subprocess285import tempfile286import os287import shutil288from typing import Optional, List289 290class Sandbox:291 def __init__(self, task: dict):292 self.task = task293 self.tmpdir: Optional[str] = None294 self.file_tree: List[str] = []295 296 def setup(self):297 """Clone repo at the specific SHA. Called by env.reset()."""298 self.tmpdir = tempfile.mkdtemp(prefix="flakysleuth_")299 try:300 # Shallow clone for speed301 subprocess.run([302 "git", "clone", "--depth=50",303 self.task["repo_url"],304 self.tmpdir305 ], capture_output=True, timeout=60, check=True)306 307 # Checkout exact SHA where flakiness was detected308 subprocess.run([309 "git", "checkout", self.task["sha"]310 ], cwd=self.tmpdir, capture_output=True, timeout=30, check=True)311 312 self.file_tree = self._build_file_tree()313 except Exception as e:314 self.cleanup()315 raise RuntimeError(f"Sandbox setup failed: {e}")316 317 def read_file(self, relative_path: str) -> Optional[str]:318 """Read a file relative to repo root. Returns None if not found."""319 full_path = os.path.normpath(os.path.join(self.tmpdir, relative_path))320 # Security: ensure path stays inside tmpdir321 if not full_path.startswith(self.tmpdir):322 return None323 if not os.path.isfile(full_path):324 return None325 try:326 with open(full_path, "r", errors="replace") as f:327 return f.read()[:4000] # cap to avoid huge files328 except Exception:329 return None330 331 def grep(self, pattern: str) -> str:332 """Grep for pattern across all .py files in the repo."""333 if not self.tmpdir:334 return "ERROR: Sandbox not initialized"335 try:336 result = subprocess.run(337 ["grep", "-rn", "--include=*.py", pattern, "."],338 cwd=self.tmpdir,339 capture_output=True,340 text=True,341 timeout=10342 )343 output = result.stdout[:2000]344 return output if output else f"No matches found for: {pattern}"345 except subprocess.TimeoutExpired:346 return "Search timed out"347 except Exception as e:348 return f"Search error: {e}"349 350 def run_test(self, pytest_test_name: str) -> str:351 """352 Run the specific test via pytest.353 ONLY called for non-OD tasks.354 """355 if self.task["category"] in ("OD", "OD-Brit", "OD-Vic"):356 return (357 "Test execution skipped for order-dependent tests. "358 "Use read_file and search_code to analyze static code structure instead. "359 "Look for: shared state, missing setUp/tearDown, module-scoped fixtures, global mutations."360 )361 try:362 result = subprocess.run(363 ["python", "-m", "pytest", pytest_test_name,364 "--tb=short", "-x", "--timeout=30", "-q"],365 cwd=self.tmpdir,366 capture_output=True,367 text=True,368 timeout=60369 )370 output = (result.stdout + result.stderr)[:2000]371 return output if output else "Test completed with no output"372 except subprocess.TimeoutExpired:373 return "Test execution timed out (>60s)"374 except Exception as e:375 return f"Test execution error: {e}"376 377 def cleanup(self):378 """Remove temp directory. Called after episode ends."""379 if self.tmpdir and os.path.exists(self.tmpdir):380 shutil.rmtree(self.tmpdir, ignore_errors=True)381 self.tmpdir = None382 self.file_tree = []383 384 def _build_file_tree(self) -> List[str]:385 """Return top-2-level file paths relative to repo root."""386 result = []387 for root, dirs, files in os.walk(self.tmpdir):388 # Skip hidden dirs and common noise389 dirs[:] = [d for d in dirs if not d.startswith(".")390 and d not in ("node_modules", "__pycache__", ".git", "venv", ".tox")]391 depth = root.replace(self.tmpdir, "").count(os.sep)392 if depth <= 2:393 for f in files:394 rel = os.path.relpath(os.path.join(root, f), self.tmpdir)395 result.append(rel)396 if len(result) > 100:397 break398 return result[:100]399```400 401---402 403## 5. Task Loader (`env/task_loader.py`)404 405```python406import pandas as pd407import random408from typing import Optional409 410class TaskLoader:411 def __init__(self, csv_path: str):412 df = pd.read_csv(csv_path)413 # Expand task_types column into individual rows414 rows = []415 for _, row in df.iterrows():416 for tt in str(row["task_types"]).split(";"):417 r = row.to_dict()418 r["task_type"] = tt.strip()419 rows.append(r)420 self.tasks = rows421 self._forced_type: Optional[str] = None422 423 def sample(self) -> dict:424 """Sample a random task, optionally filtered by type."""425 pool = self.tasks426 if self._forced_type:427 pool = [t for t in self.tasks if t["task_type"] == self._forced_type]428 task = random.choice(pool).copy()429 task["task_description"] = self._make_description(task)430 return task431 432 def force_task_type(self, task_type: str):433 """Force next sample() calls to return a specific task type."""434 self._forced_type = task_type435 436 def _make_description(self, task: dict) -> str:437 tt = task["task_type"]438 if tt == "classify":439 return (440 "Investigate the given test and determine whether it is FLAKY or STABLE. "441 "Use read_file and search_code to gather evidence. "442 "When confident, call classify_flakiness with argument 'flaky' or 'stable'."443 )444 elif tt == "root_cause":445 return (446 f"This test is confirmed flaky. Identify its root cause category. "447 f"Valid categories: OD, OD-Brit, OD-Vic, NIO, NOD, TD, TZD, ID, NDOI. "448 f"Use read_file and search_code to find evidence. "449 f"Call classify_root_cause with the category code when confident."450 )451 elif tt == "fix_proposal":452 return (453 f"This test is confirmed flaky with root cause: {task['category']}. "454 f"Propose a concrete fix as a unified diff. "455 f"Use read_file and search_code to understand the code. "456 f"Call propose_fix with a valid unified diff string."457 )458 return "Investigate the flaky test."459```460 461---462 463## 6. Core Environment (`env/environment.py`)464 465```python466import random467from env.models import FlakySleuthObservation, FlakySleuthAction468from env.sandbox import Sandbox469from env.task_loader import TaskLoader470from graders import grade_action471 472FLAKY_SIGNAL_PATTERNS = [473 "sleep", "random", "time", "datetime", "thread", "asyncio",474 "fixture", "setUp", "tearDown", "global", "shared", "singleton",475 "os.environ", "socket", "timeout", "retry", "mock", "patch"476]477 478class FlakySleuthEnv:479 def __init__(self, dataset_path: str = "dataset/py_tasks.csv"):480 self.loader = TaskLoader(dataset_path)481 self.sandbox: Sandbox = None482 self.current_task: dict = None483 self.step_count: int = 0484 self.cumulative_progress: float = 0.0485 self.files_read: set = set()486 self.episode_actions: list = []487 488 def reset(self) -> FlakySleuthObservation:489 # Cleanup previous episode490 if self.sandbox:491 self.sandbox.cleanup()492 493 # Sample new task494 self.current_task = self.loader.sample()495 self.sandbox = Sandbox(self.current_task)496 self.sandbox.setup()497 498 # Reset episode state499 self.step_count = 0500 self.cumulative_progress = 0.0501 self.files_read = set()502 self.episode_actions = []503 504 return self._make_obs()505 506 def step(self, action: FlakySleuthAction):507 self.step_count += 1508 self.episode_actions.append(action)509 tool_output = None510 reward = 0.0511 done = False512 info = {}513 514 TERMINAL_ACTIONS = ("classify_flakiness", "classify_root_cause", "propose_fix")515 516 if action.action_type in TERMINAL_ACTIONS:517 # Grade terminal action518 terminal_score = grade_action(action, self.current_task)519 520 # Late step penalty: -0.05 per step beyond 15521 late_penalty = max(0, (self.step_count - 15)) * 0.05522 523 # Wrong-direction penalty for T1524 wrong_dir_penalty = 0.0525 if (action.action_type == "classify_flakiness"526 and action.argument.lower() == "stable"527 and self.current_task.get("label") == "flaky"):528 wrong_dir_penalty = 0.2529 530 reward = min(0.999, max(0.001,531 self.cumulative_progress + terminal_score532 - late_penalty - wrong_dir_penalty533 ))534 done = True535 info = {536 "terminal_score": terminal_score,537 "progress_score": self.cumulative_progress,538 "late_penalty": late_penalty,539 "task_type": self.current_task["task_type"],540 "category": self.current_task["category"],541 }542 543 else:544 # Exploratory action545 tool_output, progress = self._execute_exploration(action)546 self.cumulative_progress = min(0.30, self.cumulative_progress + progress)547 reward = progress548 549 obs = self._make_obs(tool_output)550 return obs, reward, done, info551 552 def state(self) -> dict:553 return {554 "repo_url": self.current_task["repo_url"] if self.current_task else None,555 "test_name": self.current_task["test_name"] if self.current_task else None,556 "task_type": self.current_task["task_type"] if self.current_task else None,557 "step_count": self.step_count,558 "files_read": list(self.files_read),559 "cumulative_progress": self.cumulative_progress,560 }561 562 def _execute_exploration(self, action: FlakySleuthAction):563 progress = 0.0564 output = ""565 566 if action.action_type == "read_file":567 content = self.sandbox.read_file(action.argument)568 if content is None:569 output = f"ERROR: File not found: {action.argument}"570 progress = -0.05 # hallucination penalty571 elif action.argument in self.files_read:572 output = content573 progress = 0.0 # no reward for re-read574 else:575 self.files_read.add(action.argument)576 output = content577 progress = self._file_relevance_reward(action.argument)578 579 elif action.action_type == "search_code":580 output = self.sandbox.grep(action.argument)581 progress = self._search_relevance_reward(action.argument)582 583 elif action.action_type == "run_test":584 output = self.sandbox.run_test(self.current_task["test_name"])585 # Reward for actually running the test (shows initiative)586 # But 0 if OD task (sandbox returns static message)587 if self.current_task["category"] not in ("OD", "OD-Brit", "OD-Vic"):588 progress = 0.05589 590 return output, progress591 592 def _file_relevance_reward(self, filepath: str) -> float:593 task = self.current_task594 test_file = task.get("test_file", "")595 596 if test_file and test_file in filepath:597 return 0.0017 # reading the actual test file598 if any(filepath.endswith(ext) for ext in (".py",)):599 return 0.0013 # any python file600 return 0.0011 # non-python file (requirements, config, etc.)601 602 def _search_relevance_reward(self, pattern: str) -> float:603 pattern_lower = pattern.lower()604 if any(sig in pattern_lower for sig in FLAKY_SIGNAL_PATTERNS):605 return 0.0014 # searching for known flakiness signals606 return 0.0011 # generic search607 608 def _make_obs(self, tool_output=None) -> FlakySleuthObservation:609 task = self.current_task610 return FlakySleuthObservation(611 repo_url=task["repo_url"],612 test_name=task["test_name"],613 test_code=task.get("test_code", "")[:2000],614 file_tree=self.sandbox.file_tree if self.sandbox else [],615 tool_output=tool_output,616 task_type=task["task_type"],617 task_description=task["task_description"],618 step_count=self.step_count,619 )620```621 622---623 624## 7. Graders625 626### 7.1 Dispatcher (`graders/__init__.py`)627 628```python629from env.models import FlakySleuthAction630from graders.task1_grader import grade as grade_t1631from graders.task2_grader import grade as grade_t2632from graders.task3_grader import grade as grade_t3633 634def grade_action(action: FlakySleuthAction, task: dict) -> float:635 tt = task["task_type"]636 if tt == "classify":637 return grade_t1(action, task)638 elif tt == "root_cause":639 return grade_t2(action, task)640 elif tt == "fix_proposal":641 return grade_t3(action, task)642 return 0.001643```644 645### 7.2 Task 1 Grader (`graders/task1_grader.py`)646 647```python648from env.models import FlakySleuthAction649 650def grade(action: FlakySleuthAction, task: dict) -> float:651 """Binary classification: flaky or stable. Exact match only."""652 if action.action_type != "classify_flakiness":653 return 0.001654 655 predicted = action.argument.strip().lower()656 if predicted not in ("flaky", "stable"):657 return 0.001658 659 # All IDoFT rows are flaky; stable examples are synthetically added660 # with label="stable" during dataset construction661 ground_truth = task.get("label", "flaky")662 return 0.999 if predicted == ground_truth else 0.0663```664 665### 7.3 Task 2 Grader (`graders/task2_grader.py`)666 667```python668import json669import os670from env.models import FlakySleuthAction671 672# Load similarity matrix once at module level673_SIM_PATH = os.path.join(os.path.dirname(__file__), 674 "..", "dataset", "category_similarity.json")675with open(_SIM_PATH) as f:676 _RAW_SIM = json.load(f)677 678def _get_similarity(pred: str, true: str) -> float:679 if pred == true:680 return 0.999681 key1 = f"{pred},{true}"682 key2 = f"{true},{pred}"683 return _RAW_SIM.get(key1, _RAW_SIM.get(key2, 0.0))684 685VALID_CATEGORIES = {686 "OD", "OD-Brit", "OD-Vic", "NIO", "NOD",687 "UD", "TD", "TZD", "ID", "NDOI", "NDOD", "OSD"688}689 690def grade(action: FlakySleuthAction, task: dict) -> float:691 """692 Root cause category classification.693 Exact match = 1.0694 Related category = partial credit via similarity matrix695 Wrong family = 0.0696 """697 if action.action_type != "classify_root_cause":698 return 0.001699 700 predicted = action.argument.strip().upper()701 702 # Handle common variations703 predicted = predicted.replace(" ", "-") # "OD Brit" → "OD-Brit"704 705 if predicted not in VALID_CATEGORIES:706 return 0.001 # invalid category string707 708 # Take primary category from dataset (first if semicolon-separated)709 true_category = str(task.get("category", "")).split(";")[0].strip().upper()710 711 return _get_similarity(predicted, true_category)712```713 714### 7.4 Task 3 Grader (`graders/task3_grader.py`)715 716```python717import subprocess718import tempfile719import os720import json721from openai import OpenAI722from env.models import FlakySleuthAction723 724CATEGORY_DESCRIPTIONS = {725 "TD": "Time-Dependent: test fails due to reliance on wall-clock time",726 "TZD": "Timezone-Dependent: test fails in different timezones",727 "NOD": "Non-Deterministic: test fails due to randomness or non-determinism",728 "NIO": "Non-Idempotent-Outcome: test passes first run but fails on second run",729 "ID": "Implementation-Dependent: test fails due to language/runtime non-determinism (e.g. dict ordering)",730}731 732EXPECTED_FIX_PATTERNS = {733 "TD": ["freeze_time", "mock", "patch", "utcnow", "datetime", "monkeypatch"],734 "TZD": ["timezone", "utc", "pytz", "zoneinfo", "tzinfo", "UTC"],735 "NOD": ["seed", "mock", "patch", "deterministic", "sorted"],736 "NIO": ["setUp", "tearDown", "fixture", "yield", "cleanup", "autouse"],737 "ID": ["sorted(", "list(", "frozenset", "OrderedDict"],738}739 740def grade(action: FlakySleuthAction, task: dict) -> float:741 """742 Fix proposal grader.743 Component A: Pattern check — 0.35 weight744 Component B: Diff applies — 0.25 weight 745 Component C: LLM judge — 0.40 weight746 """747 if action.action_type != "propose_fix":748 return 0.001749 750 proposed_fix = action.argument.strip()751 if not proposed_fix:752 return 0.001753 754 category = str(task.get("category", "")).split(";")[0].strip().upper()755 known_fix = task.get("known_fix_diff", "") or ""756 test_code = task.get("test_code", "") or ""757 758 # ── Component A: Pattern check ────────────────────────────────759 patterns = EXPECTED_FIX_PATTERNS.get(category, [])760 if patterns:761 matches = sum(1 for p in patterns if p in proposed_fix)762 pattern_score = min(0.999, matches / max(1, len(patterns) * 0.4))763 else:764 pattern_score = 0.5765 766 # ── Component B: Diff applies cleanly ─────────────────────────767 apply_score = _check_diff_applies(proposed_fix, task)768 769 # ── Component C: LLM judge ────────────────────────────────────770 judge_score = _llm_judge(proposed_fix, known_fix, category, test_code)771 772 total = (0.35 * pattern_score) + (0.25 * apply_score) + (0.40 * judge_score)773 return round(min(0.999, max(0.001, total)), 4)774 775 776def _check_diff_applies(fix: str, task: dict) -> float:777 """Try a dry-run patch application against the test file in a temp copy."""778 try:779 test_file = task.get("test_file", "")780 sandbox_path = task.get("sandbox_test_path", "")781 782 if not sandbox_path or not os.path.exists(sandbox_path):783 return 0.3 # can't verify, neutral-ish784 785 with tempfile.NamedTemporaryFile(mode="w", suffix=".patch", delete=False) as f:786 f.write(fix)787 patch_path = f.name788 789 result = subprocess.run(790 ["patch", "--dry-run", "-p1", sandbox_path, patch_path],791 capture_output=True, text=True, timeout=10792 )793 os.unlink(patch_path)794 return 0.999 if result.returncode == 0 else 0.0795 except Exception:796 return 0.3 # can't verify, neutral797 798 799def _llm_judge(proposed: str, known: str, category: str, test_code: str) -> float:800 """Call the LLM judge via OpenAI-compatible API."""801 client = OpenAI(802 api_key=os.environ.get("OPENAI_API_KEY", ""),803 base_url=os.environ.get("API_BASE_URL", "https://api.openai.com/v1"),804 )805 model = os.environ.get("MODEL_NAME", "gpt-4o-mini")806 807 cat_desc = CATEGORY_DESCRIPTIONS.get(category, f"Flakiness category: {category}")808 known_section = f"Known accepted fix (from merged PR):\n```\n{known[:800]}\n```" if known else "Known fix: Not available"809 810 prompt = f"""You are evaluating a proposed fix for a flaky Python test.811 812Flakiness category: {category}813What this means: {cat_desc}814 815Original flaky test code:816```python817{test_code[:1000]}818```819 820Proposed fix (unified diff):821```822{proposed[:1000]}823```824 825{known_section}826 827Score the proposed fix from 0 to 10:828- 0–2: Fix is wrong, irrelevant, or makes things worse829- 3–5: Fix partially addresses the issue but misses root cause830- 6–8: Fix correctly addresses root cause with minor issues831- 9–10: Fix is correct, clean, minimal, and addresses root cause completely832 833Respond ONLY with a JSON object and nothing else:834{{"score": <integer 0-10>, "reason": "<one sentence explanation>"}}"""835 836 try:837 resp = client.chat.completions.create(838 model=model,839 messages=[{"role": "user", "content": prompt}],840 max_tokens=100,841 temperature=0.0,842 )843 raw = resp.choices[0].message.content.strip()844 # Strip markdown fences if present845 raw = raw.replace("```json", "").replace("```", "").strip()846 data = json.loads(raw)847 score = int(data["score"])848 return max(0.0, min(10.0, score)) / 10.0849 except Exception:850 return 0.5 # fallback neutral on any failure851```852 853---854 855## 8. OpenEnv HTTP Server (`server.py`)856 857```python858from fastapi import FastAPI, HTTPException859from env.models import FlakySleuthObservation, FlakySleuthAction860from env.environment import FlakySleuthEnv861 862app = FastAPI(title="FlakySleuth Environment")863env = FlakySleuthEnv()864 865@app.post("/reset")866def reset() -> FlakySleuthObservation:867 return env.reset()868 869@app.post("/step")870def step(action: FlakySleuthAction):871 obs, reward, done, info = env.step(action)872 return {873 "observation": obs.dict(),874 "reward": reward,875 "done": done,876 "info": info,877 }878 879@app.get("/state")880def state():881 return env.state()882 883@app.get("/health")884def health():885 return {"status": "ok"}886 887if __name__ == "__main__":888 import uvicorn889 uvicorn.run(app, host="0.0.0.0", port=7860)890```891 892---893 894## 9. `openenv.yaml`895 896```yaml897name: flaky-sleuth-env898version: 0.1.0899description: >900 An RL environment where an LLM agent investigates flaky tests in real901 Python GitHub repositories. The agent uses tool calls to read code,902 search for patterns, and run tests — then produces a verdict (classify,903 root cause, or fix). Tasks range from binary flakiness classification904 to proposing concrete code fixes verified by a hybrid grader.905 906observation_type: FlakySleuthObservation907action_type: FlakySleuthAction908reward_range: (0.001, 0.999)909 910tasks:911 - id: task1_classify912 name: "Flaky vs. Stable Classification"913 difficulty: easy914 description: >915 Given a test from a real Python repo, classify it as flaky or stable.916 Agent must call classify_flakiness with argument 'flaky' or 'stable'.917 918 - id: task2_root_cause919 name: "Root Cause Category Identification"920 difficulty: medium921 description: >922 Given a confirmed flaky test, identify the root cause category923 (OD, NOD, TD, TZD, NIO, ID, etc.) via static code analysis.924 925 - id: task3_fix_proposal926 name: "Fix Proposal"927 difficulty: hard928 description: >929 Given a confirmed flaky test and its root cause, propose a concrete930 fix as a unified diff. Evaluated by pattern matching + LLM judge.931 932episode_max_steps: 20933baseline_script: inference.py934 935infra:936 vcpu: 2937 memory_gb: 8938 max_inference_minutes: 20939```940 941---942 943## 10. Baseline Inference Script (`inference.py`)944 945**CRITICAL:** Must be named exactly `inference.py` in the root directory. Must use OpenAI client. Must read `API_BASE_URL`, `MODEL_NAME`, `OPENAI_API_KEY` from environment variables.946 947```python948"""949FlakySleuth baseline inference script.950 951Required environment variables:952 OPENAI_API_KEY — API key953 API_BASE_URL — LLM endpoint (default: https://api.openai.com/v1)954 MODEL_NAME — Model identifier (default: gpt-4o-mini)955 956Runs 5 episodes × 3 task types = 15 total episodes.957Prints average score per task type.958Must complete in under 20 minutes on vcpu=2, 8GB RAM.959"""960 961import os962import json963from openai import OpenAI964from env.environment import FlakySleuthEnv965from env.models import FlakySleuthAction966 967# ── Configuration ──────────────────────────────────────────────────968API_KEY = os.environ.get("OPENAI_API_KEY", "")969API_BASE_URL = os.environ.get("API_BASE_URL", "https://api.openai.com/v1")970MODEL_NAME = os.environ.get("MODEL_NAME", "gpt-4o-mini")971EPISODES_PER_TASK = 5972 973client = OpenAI(api_key=API_KEY, base_url=API_BASE_URL)974 975# ── System prompt (teaches the model your tool interface) ──────────976SYSTEM_PROMPT = """You are a flaky test detective. You investigate Python tests in real GitHub repositories.977 978At each step, respond ONLY with a single valid JSON object — no explanation, no markdown, no extra text.979 980Available actions:981 982EXPLORATORY (use these to gather evidence):983{"action_type": "read_file", "argument": "relative/path/to/file.py"}984{"action_type": "search_code", "argument": "pattern_to_grep_for"}985{"action_type": "run_test", "argument": ""}986 987TERMINAL (use exactly one of these to end the episode):988{"action_type": "classify_flakiness", "argument": "flaky"}989{"action_type": "classify_flakiness", "argument": "stable"}990{"action_type": "classify_root_cause", "argument": "OD"}991{"action_type": "classify_root_cause", "argument": "NOD"}992{"action_type": "classify_root_cause", "argument": "TD"}993{"action_type": "classify_root_cause", "argument": "TZD"}994{"action_type": "classify_root_cause", "argument": "NIO"}995{"action_type": "classify_root_cause", "argument": "ID"}996{"action_type": "classify_root_cause", "argument": "OD-Brit"}997{"action_type": "classify_root_cause", "argument": "OD-Vic"}998{"action_type": "propose_fix", "argument": "--- a/path\\n+++ b/path\\n@@ ... @@\\n-old line\\n+new line"}999 1000RULES:10011. Always read the test file first before making a terminal decision.10022. Search for flakiness signals: sleep, random, time, datetime, thread, os.environ, shared state.10033. For order-dependent (OD) tests, run_test is disabled — use static analysis only.10044. Call a terminal action only when you have enough evidence.10055. Respond with ONLY valid JSON. Nothing else."""1006 1007 1008def obs_to_prompt(obs) -> str:1009 return f"""TASK: {obs.task_description}1010 1011Repository: {obs.repo_url}1012Test name: {obs.test_name}1013Step: {obs.step_count}/201014 1015Test source code:1016```python1017{obs.test_code}1018```1019 1020Repository file tree (top-level):1021{chr(10).join(obs.file_tree[:40])}1022 1023Result of your last action:1024{obs.tool_output or "(No action taken yet — this is the start of the episode)"}1025 1026What is your next action? Respond with JSON only."""1027 1028 1029def run_episode(env: FlakySleuthEnv) -> float:1030 obs = env.reset()1031 messages = [1032 {"role": "system", "content": SYSTEM_PROMPT},1033 {"role": "user", "content": obs_to_prompt(obs)},1034 ]1035 total_reward = 0.01036 1037 for step in range(20):1038 try:1039 resp = client.chat.completions.create(1040 model=MODEL_NAME,1041 messages=messages,1042 max_tokens=400,1043 temperature=0.0,1044 )1045 raw = resp.choices[0].message.content.strip()1046 messages.append({"role": "assistant", "content": raw})1047 1048 # Parse action1049 clean = raw.replace("```json", "").replace("```", "").strip()1050 action_dict = json.loads(clean)1051 action = FlakySleuthAction(**action_dict)1052 1053 except json.JSONDecodeError:1054 # Model produced non-JSON — inject correction message1055 messages.append({1056 "role": "user",1057 "content": "ERROR: Your response was not valid JSON. "1058 "Respond ONLY with a JSON object as specified."1059 })1060 continue1061 except Exception as e:1062 print(f" Step {step} error: {e}")1063 break1064 1065 obs, reward, done, info = env.step(action)1066 total_reward += reward1067 1068 if done:1069 print(f" Terminal: {action.action_type}({action.argument[:50]}) "1070 f"→ terminal={info.get('terminal_score', 0):.2f} "1071 f"progress={info.get('progress_score', 0):.2f} "1072 f"total={total_reward:.2f}")1073 break1074 1075 messages.append({"role": "user", "content": obs_to_prompt(obs)})1076 1077 return total_reward1078 1079 1080def main():1081 env = FlakySleuthEnv()1082 results = {"classify": [], "root_cause": [], "fix_proposal": []}1083 1084 for task_type in results.keys():1085 print(f"\n── Task type: {task_type} ──")1086 env.loader.force_task_type(task_type)1087 for ep in range(EPISODES_PER_TASK):1088 score = run_episode(env)1089 results[task_type].append(score)1090 print(f" Episode {ep+1}: {score:.3f}")1091 1092 print("\n══ BASELINE RESULTS ══")1093 for task_type, scores in results.items():1094 avg = sum(scores) / len(scores)1095 print(f" {task_type:15s}: avg={avg:.3f} scores={[round(s,3) for s in scores]}")1096 1097 overall = sum(s for scores in results.values() for s in scores)1098 overall /= sum(len(v) for v in results.values())1099 print(f" {'OVERALL':15s}: avg={overall:.3f}")1100 1101 1102if __name__ == "__main__":1103 main()1104```1105 1106---1107 1108## 11. Dockerfile1109 1110```dockerfile1111FROM python:3.11-slim1112 1113# Install git and patch (needed for sandbox)1114RUN apt-get update && apt-get install -y \1115 git \1116 patch \1117 && rm -rf /var/lib/apt/lists/*1118 1119WORKDIR /app1120 1121# Copy requirements first for layer caching1122COPY requirements.txt .1123RUN pip install --no-cache-dir -r requirements.txt1124 1125# Copy everything else1126COPY . .1127 1128# Expose port for HF Spaces1129EXPOSE 78601130 1131# Start FastAPI server1132CMD ["python", "server.py"]1133```1134 1135---1136 1137## 12. `requirements.txt`1138 1139```1140fastapi>=0.110.01141uvicorn>=0.27.01142pydantic>=2.0.01143openai>=1.0.01144pandas>=2.0.01145gitpython>=3.1.01146pytest>=7.0.01147pytest-timeout>=2.0.01148requests>=2.31.01149```1150 1151---1152 1153## 13. Build Order (Day-by-Day Sprint)1154 1155```1156DAY 1 — Data Foundation1157────────────────────────1158□ Clone idoft repo, inspect py-data.csv manually1159□ Run build_dataset.py offline (set GITHUB_TOKEN)1160□ Verify py_tasks.csv has rows for all 3 task types1161□ Manually inspect 5-10 rows to sanity check test_code and known_fix_diff1162□ Build category_similarity.json1163 1164DAY 2 — Core Environment1165──────────────────────────1166□ Implement env/models.py (Pydantic models)1167□ Implement env/sandbox.py (clone, read_file, grep, run_test)1168□ Test sandbox.py manually on 2-3 real repos1169□ Implement env/task_loader.py1170□ Implement env/environment.py (reset, step, state)1171□ Write a quick smoke test: reset() → 3 steps → terminal action1172 1173DAY 3 — Graders1174────────────────1175□ Implement graders/task1_grader.py1176□ Implement graders/task2_grader.py + verify similarity matrix1177□ Implement graders/task3_grader.py (pattern + diff + LLM judge)1178□ Unit test all 3 graders with hardcoded inputs1179□ Verify scores are always in (0.001, 0.999)1180 1181DAY 4 — Server + Spec Compliance1182──────────────────────────────────1183□ Implement server.py (FastAPI: /reset, /step, /state, /health)1184□ Write openenv.yaml1185□ Run openenv validate — fix any errors1186□ Build Dockerfile locally: docker build . && docker run -p 7860:78601187□ Test endpoints with curl1188 1189DAY 5 — Inference Script + Deploy1190────────────────────────────────────1191□ Implement inference.py (ReAct loop, OpenAI client)1192□ Run inference.py locally against real API1193□ Verify it completes in <20 min, produces scores for all 3 task types1194□ Deploy to Hugging Face Spaces1195□ Verify HF Space returns 200 on health check and responds to reset()1196□ Run pre-submission validation script1197 1198DAY 6 — Polish + Submit1199─────────────────────────1200□ Write README (env description, observation/action spaces, setup)