VivienB/data-quality-auditing-env

📊 Data Quality Auditing OpenEnv

An interactive, OpenEnv-compliant benchmark environment for evaluating AI agents on data quality auditing tasks.

🎯 Overview

The Data Quality Auditing OpenEnv simulates a real-world data engineering scenario where an AI agent must evaluate datasets, identify quality issues, and generate structured audit reports. The environment follows the standard OpenEnv API (reset, step, state) and provides a robust framework for testing agent capabilities in data analysis, reasoning, and reporting.

Key Features

• —Standardized API: Fully compliant with OpenEnv standards.
• —Multi-level Tasks: Datasets categorized into Easy, Medium, and Hard difficulties.
• —Comprehensive Taxonomy: Supports detection of missing values, duplicates, type mismatches, outliers, inconsistencies, and bias.
• —Reward System: Built-in rule-based grader that evaluates precision, recall, F1 score, and severity-weighted accuracy.
• —Baseline Agent: Includes a reproducible OpenAI-powered baseline agent.
• —Interactive UI: Deployable Gradio application for Hugging Face Spaces.

🏗️ Environment Design

Observation Space

The state represents the current view of the dataset and the agent's progress:

• —dataset_sample: A subset of rows from the dataset.
• —schema_info: Column names, expected data types, and constraints.
• —column_stats: Summary statistics for queried columns.
• —current_report: The ongoing list of identified issues.
• —steps_remaining: Number of actions the agent can still take.

Action Space

The agent interacts with the environment using discrete actions:

• —view_schema(): Retrieve the schema of the dataset.
• —sample_rows(n): View n random rows from the dataset.
• —query_column(col_name): Get summary statistics or unique values for a specific column.
• —report_issue(issue_type, column, description, severity): Log an identified issue into the audit report.
• —submit_audit(): Finalize the audit and terminate the episode.

Reward Structure

The reward incentivizes accurate and comprehensive auditing while penalizing false positives and inefficiency:

• —+10 for correctly identifying a valid issue (True Positive).
• —-5 for reporting a non-existent issue (False Positive).
• —-1 for each action taken (step penalty to encourage efficiency).
• —+50 bonus for finding all critical issues in the dataset.

🚀 Getting Started

Prerequisites

• —Python 3.11+
• —OpenAI API Key (for the baseline agent and LLM grader)

Installation

1. 1.Clone the repository and navigate to the project directory:

   git clone https://github.com/yourusername/data-quality-openenv.git
   cd data-quality-openenv

1. 1.Install the required dependencies:

   pip install -r requirements.txt

1. 1.Set your OpenAI API key:

   export OPENAI_API_KEY="your-api-key-here"

🤖 Running the Baseline Agent

The baseline agent uses the OpenAI API to systematically explore the dataset, analyze the data, and generate an audit report.

Run the agent on a specific task:

python baseline/run_agent.py --task easy_001

Run the agent on a harder task using a specific model:

python baseline/run_agent.py --task hard_001 --model gpt-4.1-mini

Run the agent on all available tasks and save the results:

python baseline/run_agent.py --all --output results.json

🧪 Evaluation & Grading

The environment includes a robust grading system (grader.py) that evaluates the agent's audit report against the ground truth issues defined in the task files.

The grader calculates:

• —Precision: The proportion of reported issues that are correct.
• —Recall: The proportion of ground truth issues that were successfully identified.
• —F1 Score: The harmonic mean of precision and recall.
• —Weighted Score: A score based on the severity of the identified issues (Critical=4, High=3, Medium=2, Low=1).
• —Letter Grade: A final grade (A-F) based on the normalized weighted score.

You can run the grader standalone:

python grader.py --report path/to/report.json --ground-truth path/to/ground_truth.json

📦 Deployment (Hugging Face Spaces)

The project includes a Gradio web application (app.py) that provides an interactive UI for exploring tasks and running the baseline agent.

To run the app locally:

python app.py

Then open your browser to http://localhost:7860.

To deploy via Docker:

docker build -t data-quality-openenv .
docker run -p 7860:7860 data-quality-openenv

📁 Project Structure

data_quality_openenv/
├── app.py                  # Gradio UI for Hugging Face Spaces
├── baseline/
│   └── run_agent.py        # OpenAI-powered baseline agent
├── Dockerfile              # Docker configuration for deployment
├── env.py                  # Core OpenEnv environment implementation
├── grader.py               # Reward and grading system
├── models.py               # Pydantic typed data models
├── problem_definition.md   # Detailed problem scope and design document
├── README.md               # Project documentation
├── requirements.txt        # Python dependencies
└── tasks/                  # Task datasets and ground truth
    ├── easy.json
    ├── medium.json
    └── hard.json

📝 License

This project is licensed under the MIT License. See the LICENSE file for details.