jaydeepshah2025/data-quality-env

DataQual — Data Quality Validation Environment

Train AI agents to do what data engineers do every day — find and fix broken data before it breaks everything downstream.

An OpenEnv environment where AI agents audit datasets to identify and fix real-world data quality issues. Unlike toy environments, DataQual models a task that costs enterprises $12.9M/year (Gartner) — making it directly useful for evaluating and training production-grade data agents.

Why Data Quality?

Data quality problems are the silent killer of ML pipelines, analytics dashboards, and business decisions. Manual auditing is slow, error-prone, and doesn't scale. This environment lets you train and evaluate AI agents on the exact task data engineers and analysts perform daily — catching bad emails, missing fields, duplicate rows, arithmetic mismatches, and logical contradictions across columns.

Real-world skills tested:

• —Pattern recognition (email/phone/date validation)
• —Cross-record deduplication
• —Type-aware null detection
• —Multi-field logical reasoning
• —Arithmetic verification

What Makes This Different?

Unlike traditional data quality tools that rely on static rules, DataQual is designed as a reinforcement learning environment.

• —Agents must discover issues, not just validate predefined rules
• —Rewards are incremental and structured, enabling learning over time
• —Tasks require multi-step reasoning, not single-pass validation
• —Supports training, evaluation, and benchmarking of data agents

This makes DataQual suitable for building next-generation AI systems that can autonomously audit and improve data quality in production pipelines.

Action & Observation Spaces

Action

The agent sends a JSON message containing identified issues:

{
  "findings": [
    {
      "record_id": 2,
      "field": "email",
      "issue_type": "format_error",
      "description": "Double @ symbol in email address",
      "suggested_fix": "alice.johnson@gmail.com"
    }
  ]
}

Valid issue types: format_error · missing_value · duplicate_record · type_mismatch · inconsistency · invalid_range

Observation

Formatted text containing the task description, data records (JSON), grading feedback from the previous step, and remaining step count.

Tasks

Difficulty Progression

• —Easy — straightforward pattern matching (regex-level checks)
• —Medium — requires cross-record comparison and type awareness
• —Hard — multi-field reasoning, domain knowledge, arithmetic verification

Reward Function

Each correctly identified issue earns a fraction of the total score (normalised to 0.0-1.0):

Reward is incremental — each step awards points only for newly discovered issues, providing meaningful signal over the full trajectory. This means agents learn to be thorough (find all issues) while being precise (avoid false positives).

Setup & Usage

Prerequisites

• —Python 3.11+
• —Docker (for containerised deployment)

Install from HF Spaces

pip install git+https://huggingface.co/spaces/jaydeepshah2025/data-quality-env

Local Development

pip install -r requirements.txt
uvicorn server.app:app --host 0.0.0.0 --port 7860
# Server starts at http://localhost:7860

Docker

docker build -t data-quality-env .
docker run -p 7860:7860 data-quality-env

Python Client

from client import DataQualEnv
from models import DataQualAction

with DataQualEnv(base_url="http://localhost:7860").sync() as env:
    result = env.reset(task_name="format_validation")
    print(result.observation.content)

    result = env.step(DataQualAction(message='{"findings": [...]}'))
    print(f"Reward: {result.reward}, Done: {result.done}")

    state = env.state()
    print(f"Score: {state.current_score}")

Running the Baseline Inference

export API_BASE_URL="https://your-llm-api/v1"
export MODEL_NAME="your-model-name"
export HF_TOKEN="your-api-key"
export ENV_URL="http://localhost:7860"
python inference.py

API Endpoints

Baseline Scores

Approximate scores using GPT-4o-mini (temperature 0.1):

Project Structure

├── server/
│   ├── __init__.py        Server package
│   ├── app.py             FastAPI server (HTTP + WebSocket)
│   └── environment.py     Core environment logic (reset / step / state)
├── models.py              Pydantic models (Action, Observation, State)
├── tasks.py               Task data definitions and known issues
├── client.py              HTTP client for the environment
├── inference.py           Baseline inference script
├── openenv.yaml           OpenEnv metadata
├── pyproject.toml         Package definition (pip-installable)
├── Dockerfile             Container definition
├── requirements.txt       Python dependencies
└── README.md              This file

Live Demo

The environment is deployed and running at:

https://huggingface.co/spaces/jaydeepshah2025/data-quality-env

curl https://jaydeepshah2025-data-quality-env.hf.space/health
# {"status": "healthy"}