CoolFace
Apppublic

PushkarDesai/openenv-etl

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

ETL Pipeline Fixer (OpenEnv)

This project implements a custom OpenEnv environment for diagnosing and fixing broken ETL pipelines.

Objective

An agent must repair a broken pipeline.py by using three actions:

  • —edit_code(filepath, target_line, new_code)
  • —query_db(sql_string, max_rows=50)
  • —run_pipeline(timeout_override_s=30.0)

Observation fields are text-based and designed for software-debugging workflows:

  • —console_logs
  • —current_script_content
  • —db_state_summary

Reward is in [0.0, 1.0], where 1.0 means the task is fully solved.

Difficulty Levels

Easy: Schema mismatch

  • —Source CSV uses user_id
  • —Target table expects id
  • —Agent must fix the INSERT column mapping

Medium: Silent data loss

  • —Script uses INNER JOIN for users and purchases
  • —Users without purchases are dropped
  • —Agent must change query to LEFT JOIN

Hard: Concurrency race

  • —Multiple threads insert into SQLite without synchronization
  • —Fails with database is locked or drops inserts
  • —Agent must use threading.Lock around write path to reach full ingestion

Core Files

  • —env.py: OpenEnv environment implementation (reset, step, state)
  • —models.py: Pydantic action and observation models
  • —server/app.py: FastAPI/OpenEnv server wiring
  • —server/etl_environment.py: server adapter for ETLEnv
  • —client.py: OpenEnv client helper
  • —../baseline.py: Gemini tool-calling baseline runner

Run Locally

From the repository root:

bash
python test_env_phase2.py
python test_env_phase3.py

Run OpenEnv validation from this folder:

bash
openenv validate .

Baseline Agent

The baseline runner is at ../baseline.py.

bash
set GEMINI_API_KEY=<your_key>
python baseline.py

It evaluates easy, medium, and hard and prints final rewards.

Docker (Repository Root)

A root-level Dockerfile and .dockerignore are included for rubric-style docker build and docker run checks.

bash
docker build -t openenv-etl .
docker run --rm -it -e GEMINI_API_KEY=%GEMINI_API_KEY% openenv-etl

Hugging Face Space Deployment

From this folder (etl/):

bash
openenv validate .
openenv push --repo-id <username>/openenv-etl

Space app entrypoint is defined in openenv.yaml as server.app:app.