Atharva9978/DataForge-Env

DataForge-Env

DataForge-Env is a real-world OpenEnv environment for evaluating agent behavior on a core frontier-ML workflow: training-data curation. Instead of browsing a web page or playing a toy game, the agent must inspect files, write Python transformations, and submit cleaned outputs for deterministic grading.

Why this environment matters

LLM teams regularly need to:

• —redact leaked PII before training
• —remove near-duplicate data that wastes budget and increases memorization risk
• —detect benchmark contamination that invalidates evaluation claims

DataForge-Env packages those workflows into a lightweight, reproducible environment that fits the competition runtime limits while still modeling genuine ML data operations work.

OpenEnv interface

The environment implements the standard OpenEnv reset(), step(), and state interface with typed models.

Action model: DataForgeAction

• —read_file
• —execute_python
• —submit_pipeline
• —noop

Observation model: DataForgeObservation

• —task_level
• —task_id
• —difficulty
• —goal
• —workspace_files
• —step_count
• —max_steps
• —progress_score
• —progress_hint
• —last_stdout
• —last_stderr
• —last_action_error
• —inherited OpenEnv fields: reward, done, metadata

Tasks

Task 1: PII purge

Difficulty: easy

The workspace contains JSONL shards with leaked email addresses, phone numbers, and HTML boilerplate. The agent must clean the text while preserving schema and all required rows.

Grader:

• —checks for email and phone removal
• —checks for HTML removal
• —checks schema preservation
• —checks row preservation

Task 2: near-deduplication

Difficulty: medium

The workspace contains multiple shards with near-duplicate rows spread across files. The agent must keep one canonical representative per duplicate cluster and preserve all unique rows.

Grader:

• —checks recall of canonical and unique rows
• —checks removal of duplicate variants
• —checks precision to avoid over-deletion

Task 3: benchmark decontamination

Difficulty: hard

The workspace contains synthetic training rows, including a hidden subset with exact benchmark-style leakage. The agent must remove leaked rows or leaked text while preserving unrelated clean rows.

Grader:

• —checks remaining leakage
• —checks clean-row preservation

Reward design

The environment uses deterministic shaped rewards throughout the trajectory:

• —small step penalty discourages loops
• —failed file access and broken Python are penalized
• —successful Python execution gives a small positive signal
• —non-submit actions get progress-delta shaping from the deterministic grader
• —submit_pipeline returns the task score strictly within (0.0, 1.0)

This creates informative trajectory feedback while keeping final evaluation reproducible.

Project structure

dataforge/
  env.py
  generator.py
  graders.py
  models.py
  sandbox.py
server/
  app.py
openenv.yaml
inference.py
Dockerfile
README.md

Running locally

Validate the environment:

openenv validate

Build and run the container:

docker build -t dataforge-env .
docker run -p 7860:7860 dataforge-env

Quick endpoint checks:

curl -X POST http://localhost:7860/reset -H "Content-Type: application/json" -d "{}"
curl -X POST http://localhost:7860/step -H "Content-Type: application/json" -d "{\"action\":{\"action_type\":\"noop\"}}"
curl http://localhost:7860/state

Baseline inference

The required inference script is inference.py. It uses the OpenAI client and reads:

• —HF_TOKEN or API_KEY
• —API_BASE_URL
• —MODEL_NAME

Example:

export HF_TOKEN=...
export API_BASE_URL=https://router.huggingface.co/v1
export MODEL_NAME=meta-llama/Meta-Llama-3-70B-Instruct
python inference.py

The script runs all 3 tasks independently and emits the required structured logs:

• —[START]
• —[STEP]
• —[END]

Submission workflow

Use this order before pushing to Hugging Face Spaces:

1. 1.Build the image with docker build -t dataforge-env .
2. 2.Run the container and confirm http://localhost:7860/health responds.
3. 3.Run python scripts/validate_submission.py to verify imports, endpoint wiring, and a smoke test of the environment.
4. 4.Set HF_TOKEN or API_KEY, then run python inference.py and confirm the log stream contains [START], [STEP], and [END] for all three tasks.

Troubleshooting

If inference exits immediately, check that HF_TOKEN or API_KEY is set and that API_BASE_URL points at a reachable OpenAI-compatible endpoint.

If Docker fails during build verification, run python scripts/validate_submission.py locally first; it reports the exact failing check.

Baseline behavior

The baseline is intentionally simple and conservative. It prompts the model to read the brief, inspect the workspace, write Python transformations, and submit when progress looks sufficient. Because the environment and graders are deterministic, baseline runs are reproducible for a fixed model and API endpoint.