Nihar-776/Hackathon-MetaPytorch
0
1# =============================================================================2# openenv.yaml — OpenEnv specification for the Record Repair environment3#4# This file is the machine-readable contract for the environment.5# The automated evaluator parses this file to:6# 1. Discover available tasks and their configurations7# 2. Validate /reset and /step request/response schemas8# 3. Confirm OpenEnv spec compliance before human judging9# 4. Verify reward values are in the declared range10#11# Do NOT modify field names or types without updating app.py to match.12# =============================================================================13 14name: record-repair-env15version: "1.0"16 17description: >18 An OpenEnv-compliant reinforcement learning environment where an LLM agent19 receives a corrupted JSON employee record and must return a fully corrected20 version in each step. Records contain 7 fields (name, email, phone, dob,21 salary, department, join_date) and are corrupted using five strategies:22 typo, null, format, swap, and numeric. The environment models a real-world23 data quality problem common in ETL pipelines and CRM imports.24 25# =============================================================================26# Endpoints27# =============================================================================28 29endpoints:30 31 reset:32 path: /reset33 method: POST34 description: >35 Start a new episode. Generates a fresh employee record, applies36 corruptions according to the task config, and returns the corrupted37 record along with a unique session_id. The session_id must be included38 in every subsequent /step call.39 request_schema:40 type: object41 required:42 - task_id43 properties:44 task_id:45 type: string46 description: One of the task IDs defined in the tasks section below47 example: task1_single_typo48 seed:49 type: integer50 description: Optional RNG seed for reproducible episodes51 example: 4252 53 step:54 path: /step55 method: POST56 description: >57 Submit a corrected employee record and receive a reward signal.58 The environment scores the correction against the original clean record59 using a three-component reward function. The episode ends when all60 corrupted fields are exactly fixed or max_steps is reached.61 request_schema:62 $ref: "#/action_space"63 64 state:65 path: /state66 method: GET67 description: >68 Return the current observation for an active session without advancing69 the episode. Pass session_id as a query parameter.70 parameters:71 session_id:72 in: query73 type: string74 format: uuid75 required: true76 77 tasks:78 path: /tasks79 method: GET80 description: >81 List all available tasks with their full configuration. No request82 body required.83 84# =============================================================================85# Action space (what the agent sends to POST /step)86# =============================================================================87 88action_space:89 type: object90 required:91 - corrected_record92 - task_id93 - session_id94 properties:95 96 corrected_record:97 type: object98 description: >99 Full 7-field employee record with the agent's corrections applied.100 All 7 fields must be present. Fields the agent believes are clean101 should be returned unchanged.102 required:103 - name104 - email105 - phone106 - dob107 - salary108 - department109 - join_date110 properties:111 name:112 type: string113 description: Full name e.g. "Alice Johnson"114 example: Alice Johnson115 email:116 type: string117 format: email118 description: Valid email address119 example: alice.johnson@example.com120 phone:121 type: string122 pattern: '^\d{3}-\d{3}-\d{4}$'123 description: Phone number in XXX-XXX-XXXX format124 example: "555-123-4567"125 dob:126 type: string127 format: date128 description: Date of birth in ISO 8601 format YYYY-MM-DD129 example: "1990-06-15"130 salary:131 type: number132 format: float133 minimum: 0134 description: Annual salary as a plain float — no currency symbol, no commas135 example: 87500.0136 department:137 type: string138 description: Department name e.g. "Engineering", "Marketing", "Finance"139 example: Engineering140 join_date:141 type: string142 format: date143 description: Employment start date in ISO 8601 format YYYY-MM-DD144 example: "2018-03-01"145 146 confidence:147 type: number148 format: float149 minimum: 0.0150 maximum: 1.0151 default: 0.9152 description: >153 Agent self-reported confidence in its correction (logged for analysis,154 not used in scoring).155 156 task_id:157 type: string158 description: Must match the task_id used in the /reset call for this session159 example: task1_single_typo160 161 session_id:162 type: string163 format: uuid164 description: Must match the session_id returned by /reset165 example: 550e8400-e29b-41d4-a716-446655440000166 167# =============================================================================168# Observation space (what the environment returns from /reset and /step)169# =============================================================================170 171observation_space:172 type: object173 required:174 - task_id175 - session_id176 - step177 - corrupted_record178 - corruption_types_hint179 - num_corrupted_fields180 - fields_still_wrong181 - score_so_far182 - max_steps183 - done184 - message185 properties:186 187 task_id:188 type: string189 description: The task ID for this episode190 example: task1_single_typo191 192 session_id:193 type: string194 format: uuid195 description: Unique identifier for this episode — include in every /step call196 example: 550e8400-e29b-41d4-a716-446655440000197 198 step:199 type: integer200 minimum: 0201 description: Number of /step calls made so far in this episode (0 after /reset)202 203 corrupted_record:204 type: object205 description: >206 The employee record with corruptions applied. This is what the agent207 must fix. The schema matches action_space.corrected_record.208 properties:209 name: { type: string }210 email: { type: string }211 phone: { type: string }212 dob: { type: string }213 salary: {} # may be string or number when corrupted214 department: { type: string }215 join_date: { type: string }216 217 corruption_types_hint:218 type: array219 description: >220 Which corruption TYPES are present in this episode. Does NOT reveal221 which specific fields are corrupted — the agent must infer that.222 items:223 type: string224 enum:225 - typo226 - null227 - format228 - swap229 - numeric230 example: ["typo", "null"]231 232 num_corrupted_fields:233 type: integer234 minimum: 0235 maximum: 7236 description: Total number of fields that were corrupted in this episode237 238 fields_still_wrong:239 type: integer240 minimum: 0241 maximum: 7242 description: >243 Number of corrupted fields that are NOT yet exactly correct after the244 last /step call. Always equals num_corrupted_fields after /reset.245 246 score_so_far:247 type: number248 format: float249 minimum: 0.0250 description: Cumulative reward accumulated across all steps so far (0.0 after /reset)251 252 max_steps:253 type: integer254 minimum: 1255 description: Maximum number of /step calls allowed before the episode ends256 257 done:258 type: boolean259 description: >260 True when the episode is over — either all corrupted fields are exactly261 fixed or max_steps has been reached.262 263 message:264 type: string265 description: Human-readable summary of the episode state after each step266 267 # Reward breakdown — populated after each /step (all 0.0 after /reset)268 269 reward:270 type: number271 format: float272 minimum: 0.0273 maximum: 1.0274 description: Combined reward for the most recent step275 276 exact_accuracy:277 type: number278 format: float279 minimum: 0.0280 maximum: 1.0281 description: Fraction of corrupted fields exactly restored in the last step282 283 fuzzy_accuracy:284 type: number285 format: float286 minimum: 0.0287 maximum: 1.0288 description: Average SequenceMatcher ratio across corrupted fields in the last step289 290 no_hallucination:291 type: number292 format: float293 minimum: 0.0294 maximum: 1.0295 description: Fraction of clean fields left unchanged by the agent in the last step296 297# =============================================================================298# Reward function299# =============================================================================300 301reward_range:302 min: 0.0303 max: 1.0304 description: >305 Reward is computed per /step call as a weighted sum of three components.306 All components and the final reward are in [0.0, 1.0].307 308 components:309 310 exact_accuracy:311 weight: 0.6312 description: >313 Fraction of corrupted fields the agent restored to the exact clean value,314 after normalisation (strip, lowercase, remove commas/hyphens/currency315 suffixes/trailing .0). Only fields in the corruption mask are evaluated.316 range: [0.0, 1.0]317 318 fuzzy_accuracy:319 weight: 0.3320 description: >321 Average SequenceMatcher ratio across all corrupted fields (after322 normalisation). Gives partial credit for near-correct answers —323 a single-character typo scores ~0.93 instead of 0.0.324 range: [0.0, 1.0]325 326 no_hallucination:327 weight: 0.1328 description: >329 Fraction of clean (non-corrupted) fields the agent left unchanged.330 Penalises overcorrection — reformatting a field that was already331 correct reduces this score.332 range: [0.0, 1.0]333 334 formula: "reward = 0.6 * exact_accuracy + 0.3 * fuzzy_accuracy + 0.1 * no_hallucination"335 336# =============================================================================337# Corruption types registry338# =============================================================================339 340corruption_types:341 342 typo:343 description: >344 Character-level error injected into a string field (name, email,345 or department). One of three error subtypes is applied randomly:346 swap (adjacent characters transposed), delete (one character removed),347 or insert (one character duplicated).348 affected_fields: [name, email, department]349 severity_scaling: >350 0=easy: 1 word in 1 field.351 1=medium: 1 word in 2 fields.352 2=hard: 2 words in 2 fields.353 354 null:355 description: >356 A field is set to None or empty string (""). The agent must infer357 a plausible value from context (e.g. derive email from name).358 affected_fields: [name, email, phone, dob, salary, department, join_date]359 severity_scaling: >360 0=easy: 1 field set to None.361 1=medium: 2 fields (one None, one empty string).362 2=hard: 3 fields (mixed None and empty string).363 364 format:365 description: >366 A phone number or date field is converted to an incorrect format.367 Phone: hyphens removed, country code added, or last 4 digits scrambled.368 Date: converted to DD/MM/YYYY, "Mon DD YYYY", or YYYY/DD/MM.369 affected_fields: [phone, dob, join_date]370 severity_scaling: >371 0=easy: minor format change (remove hyphens / DD/MM/YYYY).372 1=medium: moderate change (add country code / "Jan 15 2020").373 2=hard: severe change (scrambled digits / YYYY/DD/MM).374 375 swap:376 description: >377 Two field values are exchanged. Pairs are chosen to be obviously wrong378 (cross-type swaps) e.g. name <-> email, department <-> dob, phone <-> join_date.379 Severity does not change the number of swaps (always exactly one pair).380 affected_fields: [name, email, department, dob, phone, join_date]381 severity_scaling: Always exactly one pair swapped regardless of severity.382 383 numeric:384 description: >385 The salary field is corrupted in type, unit, or scale.386 affected_fields: [salary]387 severity_scaling: >388 0=easy: currency suffix added ("87500.0 USD" — becomes a string).389 1=medium: 10x scale error (875000.0 instead of 87500.0).390 2=hard: thousands separator added ("87,500.00" — becomes a string).391 392# =============================================================================393# Tasks394# =============================================================================395 396tasks:397 398 - id: task1_single_typo399 description: >400 Easy — fix a single character-level typo in one string field.401 The agent receives a record with exactly one corrupted field402 containing a character swap, deletion, or duplication.403 difficulty: easy404 corruption_types: [typo]405 num_corruptions: 1406 severity: 0407 max_steps: 5408 success_threshold: 0.80409 410 - id: task2_multi_corrupt411 description: >412 Medium — fix three simultaneous corruptions across different field types.413 One field is nulled out, one date or phone is in the wrong format,414 and the salary field has a type or scale error. The agent must handle415 all three independently within 10 steps.416 difficulty: medium417 corruption_types: [null, format, numeric]418 num_corruptions: 3419 severity: 1420 max_steps: 10421 success_threshold: 0.70422 423 - id: task3_full_adversarial424 description: >425 Hard — all five corruption types applied simultaneously at maximum426 severity. String fields have typos, fields are nulled, formats are427 mangled, two fields are swapped, and the salary is corrupted.428 The agent must reason carefully about each field type independently429 within 15 steps.430 difficulty: hard431 corruption_types: [typo, null, format, swap, numeric]432 num_corruptions: 5433 severity: 2434 max_steps: 15435 success_threshold: 0.60436 437# =============================================================================438# Infrastructure439# =============================================================================440 441infrastructure:442 runtime: docker443 python_version: "3.11"444 framework: fastapi445 port: 7860446 max_inference_runtime_minutes: 20447 max_memory_gb: 8448 max_vcpu: 2449 450# =============================================================================451# Environment variables required for inference.py452# =============================================================================453 454required_env_vars:455 ENV_BASE_URL:456 description: Base URL of the deployed HF Space e.g. https://nihar-776-hackathon-metapytorch.hf.space457 example: http://localhost:7860458 API_BASE_URL:459 description: Base URL of the LLM API (OpenAI-compatible)460 example: https://router.huggingface.co/v1461 MODEL_NAME:462 description: Model identifier to use for inference463 example: Qwen/Qwen2.5-72B-Instruct464 HF_TOKEN:465 description: Hugging Face API token with inference permissions466 example: hf_xxxxxxxxxxxxxxxxxxxx467