Ani2-3/data-cleaning-env

Data Cleaning Environment — OpenEnv (v3)

An OpenEnv-compatible environment where AI agents learn to clean messy real-world datasets through the standard step() / reset() / state() API.

Domain: Data cleaning — a task data scientists spend ~80% of their time on. Agents must handle ambiguous data, resolve multi-source conflicts, enforce cross-field business rules, detect anomalies, and make probabilistic decisions in realistic datasets.

Why Data Cleaning?

• —Real-world utility: Every organization deals with dirty data. This fills a genuine gap in agent evaluation.
• —Rich action space: 11 action types including conflict resolution, anomaly flagging, derived value computation, undo/rollback, and validation.
• —Multi-source conflicts: Same field can have different values from different sources (CRM vs invoice vs warehouse) — agents must reason about source reliability.
• —Ambiguity & reasoning: Tasks require contextual inference (date conventions, customer identity resolution, category synonyms), not just pattern matching.
• —Data provenance: Track which source each value came from, enabling agents to make informed decisions about conflicting data.
• —Confidence scoring: Agents declare confidence in their fixes, enabling nuanced grading.
• —Alternative valid answers: Grader accepts multiple correct solutions for ambiguous cases.
• —Multi-dimensional grading: Structural accuracy, cell-level correctness, business rule compliance, anomaly detection, and consistency.
• —Anti-gaming: Includes valid outliers and over-correction detection.

Tasks

Action Space

{
  "action_type": "string",        // Required: see table below
  "row_index": "int|null",        // Row to act on
  "column": "string|null",        // Column to act on
  "new_value": "string|null",     // New value for fix_cell, fill_missing, or resolve_conflict
  "transformation": "string|null", // For standardize_column
  "reason": "string|null",        // For flag_anomaly or resolve_conflict
  "confidence": "float|null",     // 0.0-1.0 confidence in this fix
  "source_columns": "[str]|null", // For derive_value
  "metadata": "{}"                // Additional context
}

Action Types

Transformations for standardize_column

lowercase, uppercase, titlecase, strip_whitespace, format_date_iso, format_phone_e164, normalize_currency, normalize_boolean

Observation Space

{
  "data": [{"col1": "val1", ...}],
  "columns": ["col1", "col2", ...],
  "num_rows": 29,
  "visible_rows": 7,
  "detected_issues": ["Missing value at row 3, column 'price'", ...],
  "quality_score": 0.53,
  "step_count": 3,
  "max_steps": 65,
  "task_id": "full_pipeline",
  "task_description": "...",
  "message": "Fixed [3][price]: -45.5 → 45.5. Quality +0.0120",
  "done": false,
  "reward": 0.036,
  "hints": ["Dataset has 29 rows but should have 25. Look for duplicates."],
  "recommended_actions": [{"action_type": "standardize_column", "column": "status", "transformation": "lowercase"}],
  "flagged_anomalies": [{"row_index": 5, "reason": "cost >= price"}],
  "data_sources": {"CRM": "primary customer system", "SignupForm": "web registration"},
  "metadata": {}
}

Unique Mechanics (v3)

Partial Observability

Agents only see a window of 6-8 rows initially. They must use inspect to reveal more rows and discover hidden patterns. This forces intelligent exploration rather than seeing everything upfront.

Hidden Constraints

Task descriptions are intentionally vague. Business rules (e.g., "convert EUR to USD", "dates use US convention") are NOT explicitly stated — agents must discover them from data patterns. This separates pattern-following agents from reasoning agents.

Confidence-Based Grading (Game Changer)

Confidence isn't just metadata — it directly affects scoring:

• —High confidence (>0.8) + wrong fix = BIG penalty (quadratic: 0.9 conf wrong → -0.0405 penalty)
• —Low confidence (<0.3) + wrong fix = small penalty (appropriately uncertain)
• —High confidence + correct fix = bonus reward
• —This forces calibrated intelligence, not guessing

Competing Objectives (Trade-offs)

No perfect solution exists. Agents face genuine trade-offs:

• —Fix aggressively → risk over-correcting valid data (lose points)
• —Stay conservative → leave real errors unfixed (lose points)
• —Explore thoroughly → use up step budget (time pressure)
• —Act fast → miss hidden issues (quality loss)

Time-Pressure Scoring

Strong step-count pressure in grading:

• —Finishing under 40% of max steps → efficiency bonus up to +0.06
• —Using over 70% of steps → penalty up to -0.03
• —Combined with per-step cost (-0.005) creates real urgency

Multi-Source Conflicts

Data comes from multiple sources (CRM, SignupForm, inventorysystem, suppliercatalog, warehouse_scan). When sources disagree, agents must reason about which source to trust.

Data Provenance

Each observation includes data_sources showing where values originated.

Alternative Valid Answers

The grader accepts multiple correct solutions for genuinely ambiguous cases.

Over-Correction Detection

Valid outliers must be preserved. Agents that "fix" correct data lose points.

Undo Action (Rollback)

Agents can undo the last mutating action (fixcell, deleterow, fillmissing, standardizecolumn, derivevalue, resolveconflict). The undo stack holds up to 10 snapshots. This allows agents to recover from mistakes without restarting, but undo itself costs a step.

Anti-Hallucination Detection

When an agent writes a value (e.g., customer_name) that doesn't fuzzy-match any existing data or context clues, it's flagged as a potential fabrication with a penalty. This prevents LLMs from confidently inventing plausible-sounding but incorrect values.

Mid-Episode Data Drift (Hard Task)

At step 25 in the hard task, new corruptions are injected into already-cleaned data and a late-arriving record appears. This simulates real-world data pipelines where data changes while you're cleaning it. Agents must detect and re-fix drift.

Adaptive Difficulty

After 5 successful fixes, additional hidden constraints are revealed, making the task progressively harder. This rewards agents that can adapt their strategy mid-episode.

Inspect Cost & Memory Tracking

Inspecting data is not free (except the first 3 inspects). Redundant inspects (same row+column) are penalized more heavily, rewarding agents with working memory.

Grading — Multi-Dimensional (not just cell matching)

Partial credit: Near-matches (fuzzy string similarity > 0.8) and numeric near-matches (within 5-10%) receive partial scores instead of binary 0/1.

Reward Design

Dense signal throughout the episode — not binary end-of-episode.

Setup & Usage

Local Development

pip install -r requirements.txt
uvicorn server.app:app --host 0.0.0.0 --port 8000 --reload

# In another terminal
python baseline.py --url http://localhost:8000

Docker

docker build -t data-cleaning-env .
docker run -p 8001:7860 data-cleaning-env
# Then access at http://localhost:8001

Run Tests

pip install pytest
pytest tests/ -v

API Usage

# Health check
curl http://localhost:8000/health

# List tasks
curl http://localhost:8000/tasks

# Reset with a task
curl -X POST http://localhost:8000/reset \
  -H "Content-Type: application/json" \
  -d '{"task_id": "full_pipeline", "seed": 42}'

# Take an action
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action": {"action_type": "standardize_column", "column": "order_date", "transformation": "format_date_iso"}}'

# Resolve a multi-source conflict
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action": {"action_type": "resolve_conflict", "row_index": 3, "column": "customer_name", "new_value": "Acme Corp", "reason": "CRM is authoritative for customer names", "confidence": 0.9}}'

# Flag an anomaly
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action": {"action_type": "flag_anomaly", "row_index": 5, "column": "quantity", "reason": "negative quantity", "metadata": {"type": "business_rule"}}}'

# Validate business rules
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action": {"action_type": "validate"}}'

# Derive computed field
curl -X POST http://localhost:8000/step \
  -H "Content-Type: application/json" \
  -d '{"action": {"action_type": "derive_value", "row_index": 3, "column": "total"}}'

# Get grader score
curl -X POST http://localhost:8000/grader

# Run baseline
curl -X POST http://localhost:8000/baseline

Benchmark Results (v3)

Reproducible: Run POST /benchmark or python baseline.py --mode benchmark to regenerate all results.

Config: 4 tiers x 3 tasks x 5 seeds = 60+ evaluations.

Leaderboard

Model Definitions (what each baseline actually does)

random — Picks random actions (fixcell, deleterow, etc.) with random garbage values ("INVALID", "-1", "???"). Deliberately overconfident (confidence 0.7-1.0 on wrong answers). No strategy.

rule_based — 3-5 hardcoded standardize transforms (email→lowercase, phone→E.164, boolean→normalize) + exact duplicate removal by primary key + submit. No exploration, no confidence scores, no validation, no anomaly detection. 6-9 steps total.

improved — Full heuristic pipeline:

1. 1.Explore: inspect overview + inspect rows at intervals to reveal hidden data
2. 2.Standardize: all relevant column transforms (dates, names, phones, booleans)
3. 3.Deduplicate: remove exact matches by primary key (ID/SKU)
4. 4.Fix: repair negative values, fill missing with context-aware defaults
5. 5.Derive: compute total = qty x price x (1-discount) for all rows
6. 6.Detect: flag anomalies (cost > price, non-USD currency, source conflicts)
7. 7.Validate: run business rule validation
8. 8.Submit with calibrated confidence (0.4-0.9 depending on certainty) 19-49 steps total.

Oracle Ceiling (theoretical maximum)

The oracle copies ground truth directly. The gap shows how much room remains for smarter agents.

Scoring Tiers

Difficulty Calibration (easy > medium > hard)

Difficulty is correctly calibrated: easy > medium > hard for all strategic agents (random fails because it has no strategy).

Per-Task Variance (stability proof)

Key insight: improved has 100x lower variance than random (0.003 vs 0.266). Strategic agents produce consistent results.

Ablation Study (what happens when you remove features)

Key findings:

• —Dedup is the single most impactful feature (-0.121 avg when removed)
• —Anomaly flagging matters a lot (-0.051 avg) — agents that don't flag lose 5%
• —Confidence calibration matters for hard tasks (-0.038 on full_pipeline)
• —Validation has minimal impact because the improved baseline already fixes most rule violations

Failure Analysis

format_fixing (28 issues, 4 hidden constraints):

• —Hardest: date format interpretation (19), duplicates (5), inconsistent casing (4)
• —Hidden: US date convention, CRM authority, city abbreviations, name suffixes

dedup_and_missing (15 issues, 5 hidden constraints):

• —Hardest: missing values (8), duplicates (4), casing (2)
• —Hidden: weight unit inference, timestamp-based conflicts, similar-name disambiguation, SKU format, category synonyms

full_pipeline (56 issues, 7 hidden constraints):

• —Hardest: total mismatches (10), missing values (8), date format (28)
• —Hidden: currency conversion rates, cancelled order rules, volume discounts, customer name consistency, orphan handling, payment method normalization, temporal validation

What Each Baseline Misses

Running Your Own Baseline

# Rule-based (fast, ~0.60)
python baseline.py --mode rule

# Improved heuristic (~0.78)
python baseline.py --mode improved

# LLM agent with planning & memory
OPENAI_API_KEY=sk-... python baseline.py --mode llm --model gpt-4o-mini

# Full multi-seed benchmark with statistics
python baseline.py --mode benchmark --seeds 5

API Endpoints

Project Structure

├── server/
│   ├── __init__.py
│   ├── app.py              # FastAPI server with all endpoints
│   └── environment.py      # Core environment logic, tasks, grader
├── tests/
│   ├── __init__.py
│   └── test_environment.py # Pytest test suite (25+ tests)
├── __init__.py
├── openenv.yaml            # OpenEnv spec manifest
├── pyproject.toml           # Python project config
├── Dockerfile              # Container definition
├── baseline.py             # Baseline inference script (rule + LLM modes)
├── requirements.txt        # Python dependencies
└── README.md               # This file

What Makes This Environment Challenging

1. 1.Multi-source conflicts: CRM says "Acme Corp", invoice says "ACME Corporation" — which is authoritative?
2. 2.Data provenance reasoning: Agent must consider source reliability when resolving conflicts.
3. 3.Ambiguous dates: Is 01/02/2024 January 2nd or February 1st? Context clues (US region) determine the answer.
4. 4.Near-duplicate detection: "Wireles Mouse" vs "Wireless Mouse" — typos, not just exact matches.
5. 5.Category synonym resolution: "Tech", "Electrical", "Electronics" all mean the same thing.
6. 6.Cross-referential integrity: Same customer_id must always map to the same customer_name.
7. 7.Temporal validation: ship_date must be after order_date.
8. 8.Multi-currency mixing: EUR, GBP, and USD in the same dataset — must normalize.
9. 9.Valid outlier preservation: A quantity of 200 is a legitimate bulk order — don't "fix" it.
10. 10.Business rule enforcement: total = qty × price × (1-discount), cost < price, etc.
11. 11.Confidence-based decisions: Agent must calibrate confidence for ambiguous vs obvious fixes.
12. 12.Weight unit mismatches: Some weights in kg, others in lbs — must detect and convert.
13. 13.Source trust ranking: More recent data or more authoritative sources should be preferred.
14. 14.Orphaned references: Customer IDs that don't match any existing customer.
15. 15.Cancelled order rules: Cancelled orders shouldn't have ship dates.
16. 16.Adversarial Unicode: Zero-width spaces, non-breaking spaces, en-dashes, curly quotes, and fullwidth commas hidden in text fields.
17. 17.Shipping address normalization: Unstructured addresses with abbreviations, newlines, missing parts, and format inconsistencies.
18. 18.Mid-episode data drift: At step 25, new corruptions appear — tests agent adaptability to changing data.
19. 19.Anti-hallucination detection: Fabricated values not derivable from existing data are penalized.
20. 20.Undo/rollback capability: Agents must decide when to undo mistakes vs. press forward.

Hugging Face Space

Deployed at: https://huggingface.co/spaces/Ani2-3/data-cleaning-env

Tagged with openenv for discovery.