meta-scaler/procurement

1OpenEnv Incident Debugging — Grader & Score Analysis2Purpose: This document explains exactly how this project avoids the3"One or more task scores are out of range" error, and how the grading system works end-to-end.4 51. File Structure & Responsibilities6scaler/7├── openenv.yaml            ← Task + grader configuration (the spec file)8├── inference.py             ← Baseline agent that runs all 3 tasks9├── server/10│   ├── app.py               ← FastAPI server, API endpoints, grader routes11│   ├── environment.py       ← Core RL environment (reset/step/state)12│   ├── scorer.py            ← ⭐ WHERE SCORES ARE CALCULATED + CLAMPED13│   ├── parser.py            ← Parses raw text actions into structured fields14│   ├── tasks.py             ← Gold-standard answers for easy/medium/hard15│   ├── models.py            ← Pydantic typed models (Action, Observation, State)16│   └── log_generator.py     ← Procedural noise log generator17Who does what:18File	Role19openenv.yaml	Declares 3 tasks with grader.type: score and grader.endpoint: /grade/{id}20tasks.py	Stores the gold answers each task is graded against21parser.py	Extracts ROOT_CAUSE, FACTORS, FIX, SEVERITY from raw text22scorer.py	Computes the score using F1/coverage metrics, then clamps to (0.01, 0.99)23environment.py	Orchestrates reset → step → score flow24app.py	Exposes HTTP endpoints including /grade/easy, /grade/medium, /grade/hard25inference.py	Runs all 3 tasks and emits [START]/[STEP]/[END] logs262. The Full Execution Flow27Validator/Agent calls POST /reset?task_id=easy28         │29         ▼30   ┌─────────────┐31   │ environment  │ ← Loads gold answers from tasks.py32   │   .reset()   │ ← Generates procedural logs via log_generator.py33   └──────┬──────┘34          │ returns Observation (logs, context)35          ▼36Validator/Agent calls POST /step  { "raw_text": "ROOT_CAUSE: ... FACTORS: ... FIX: ... SEVERITY: ..." }37         │38         ▼39   ┌─────────────┐40   │ environment  │41   │   .step()    │42   └──────┬──────┘43          │44          ▼45   ┌─────────────┐     ┌──────────┐46   │  parser.py   │────▶️│ scorer.py │47   │ parse_action │     │ calculate │48   └─────────────┘     │ _score_   │49                        │ and_reward│50                        └─────┬────┘51                              │52                              ▼53                    ┌──────────────────┐54                    │ CLAMP HAPPENS    │55                    │ max(0.01, min(   │56                    │   0.99, raw))    │57                    └────────┬─────────┘58                             │59                             ▼60                  Returns { score, reward, done }61                             │62                             ▼63          Validator calls GET or POST /grade/easy64                             │65                             ▼66                    ┌──────────────────┐67                    │ CLAMP AGAIN      │68                    │ max(0.01, min(   │69                    │   0.99, score))  │70                    └────────┬─────────┘71                             │72                             ▼73                  Returns { "score": 0.XX }  ← always in (0, 1)743. Grader Logic — Deep Dive753.1 Where scores are computed76The scoring happens in 77scorer.py78:79 80python81def calculate_score_and_reward(parsed, task, best_score, raw_text, previous_action_text):82It computes 4 sub-scores, each between 0.0 and 1.0:83 84Component	Weight	How it's calculated85rc_score (Root Cause)	0.4	F1 between predicted and gold root cause tokens86f_score (Factors)	0.2	F1 between predicted and gold contributing factor tokens87fix_score (Fix)	0.3	Category coverage (RESTART, SCALE, OPTIMIZE, ROLLBACK)88sev_score (Severity)	0.1	Binary: 1.0 if severity matches gold, 0.0 otherwise89The raw combined score:90 91python92current_score_raw = (rc_score * 0.4 + f_score * 0.2 + fix_score * 0.3 + sev_score * 0.1)93IMPORTANT94 95This raw score CAN be exactly 0.0 or 1.0. For example:96 97If the agent submits garbage → all sub-scores are 0.0 → raw score = 0.0 ❌98If the agent gets everything perfect → all sub-scores are 1.0 → raw score = 1.0 ❌99Both of these would fail the validator.100 1013.2 The critical fix — Score Clamping (Layer 1)102Immediately after computing the raw score, 103scorer.py line 67104 clamps it:105 106python107# Calibration: ensure score stays within (0, 1) strictly108current_score = max(0.01, min(0.99, current_score_raw))109This is the primary safeguard. It transforms the score like this:110 111Raw Score	After Clamping	Valid?1120.0	0.01	✅1130.0000001	0.01	✅1140.42	0.42	✅1150.78	0.78	✅1161.0	0.99	✅1170.9999	0.99	✅1183.3 Score Clamping (Layer 2) — Grader Endpoints119Even after Layer 1, there's a second clamp in the /grade/* endpoints in 120app.py line 609121:122 123python124score = info.get("score", 0.0)125# Clamp to strict (0, 1) — validator rejects 0.0 and 1.0126score = max(0.01, min(0.99, score))127This is defense-in-depth. Even if a code path somehow bypasses Layer 1, the grader endpoint will never return 0.0 or 1.0.128 1293.4 Why two layers?130Layer	Location	Protects against131Layer 1	scorer.py:67	Raw score calculation producing 0.0 or 1.0132Layer 2	app.py:609	Edge cases like rounding, default values, or error paths returning 0.01334. Grader Endpoint Configuration1344.1 openenv.yaml declares the graders135yaml136tasks:137  easy:138    grader:139      type: score           # ← tells validator this is a score-based grader140      endpoint: /grade/easy  # ← tells validator WHERE to call141  medium:142    grader:143      type: score144      endpoint: /grade/medium145  hard:146    grader:147      type: score148      endpoint: /grade/hard1494.2 app.py implements matching endpoints150Each endpoint supports both GET and POST (some validators use GET, some POST):151 152python153@app.get("/grade/easy")154@app.post("/grade/easy")155async def grade_easy(request: Request):156    # ... parse body ...157    return await _grade("easy", action)  # → returns { "score": 0.01–0.99 }158@app.get("/grade/medium")159@app.post("/grade/medium")160async def grade_medium(request: Request):161    return await _grade("medium", action)162@app.get("/grade/hard")163@app.post("/grade/hard")164async def grade_hard(request: Request):165    return await _grade("hard", action)166Plus fallback routes for robustness:167 168python169@app.get("/grade/{task_id}")    # dynamic catch-all170@app.post("/grade/{task_id}")171@app.get("/grade")              # generic (task_id in body)172@app.post("/grade")1734.3 Why both GET and POST matter174The validator's HTTP method is unpredictable. Your friend's fix specifically noted:175 176"I also enabled both GET and POST for grader routes so validator method mismatch doesn't break grading."177 178If you only have @app.post(...) and the validator sends a GET, it returns 405 Method Not Allowed → the validator sees no grader → ❌ fails.179 1805. Why This Project Does NOT Encounter The Error181Summary of all safeguards:182#	Safeguard	Where	What it prevents1831	Score clamping max(0.01, min(0.99, ...))	scorer.py:67	Raw score = 0.0 or 1.01842	Score clamping in grader response	app.py:609	Grader endpoint returning 0.0 or 1.01853	spec_version: 1 in openenv.yaml	openenv.yaml:1	Validator not recognizing the spec1864	grader.type: score per task	openenv.yaml:18,24,30	Validator not finding grader type1875	grader.endpoint per task	openenv.yaml:19,25,31	Validator not knowing where to call1886	Dedicated per-task routes	app.py:621-649	Endpoint not existing for a task1897	GET + POST on all grader routes	app.py:621-675	HTTP method mismatch1908	Dynamic fallback route	app.py:653-661	Unexpected task_id format1919	Generic /grade fallback	app.py:665-675	Validator calling bare /grade192The exact reason it works:193Every possible path that produces a score passes through max(0.01, min(0.99, value)).194There is no code path where a score of exactly 0.0 or 1.0 can escape to the validator.195Additionally, every grader endpoint declared in openenv.yaml actually exists server-side,196responds to both GET and POST, and always returns a score in the open interval (0, 1).197 1986. How To Apply This Fix To Another Project199Step 1: Find where your score is calculated200Look for the function that produces the final score number. It might look like:201 202python203score = some_calculation(...)204return {"score": score}205Step 2: Add clamping BEFORE returning206python207score = some_calculation(...)208score = max(0.01, min(0.99, score))  # ← ADD THIS LINE209return {"score": score}210Step 3: Add clamping in your grader endpoint too211python212@app.post("/grade/easy")213async def grade_easy(...):214    score = compute_score(...)215    score = max(0.01, min(0.99, score))  # ← DEFENSE IN DEPTH216    return {"score": score}217Step 4: Make sure grader endpoints support GET and POST218python219@app.get("/grade/easy")    # ← ADD GET220@app.post("/grade/easy")   # ← KEEP POST221async def grade_easy(...):222Step 5: Verify your openenv.yaml223yaml224tasks:225  your_task:226    grader:227      type: score              # ← must be "score"228      endpoint: /grade/your_task  # ← must match an actual route229CAUTION230 231The most common mistake is clamping in only one place. If your scoring function returns 0.0232and your grader endpoint doesn't clamp, the validator sees 0.0 and rejects it.233Always clamp in both places.