b4rty/torchdebug-env

🔥 TorchDebug — PyTorch Training Run Debugger

An OpenEnv environment that challenges AI agents to diagnose and fix real-world PyTorch training failures.

  

🎯 What is TorchDebug?

TorchDebug simulates the real work of an ML engineer debugging broken training runs. A training run is presented with:

• —📊 Training logs showing loss/accuracy/gradient progression
• —💻 Code snippets containing one or more bugs
• —⚙️ Configuration detailing hyperparameters and setup
• —❌ Error messages (if the run crashed)

The agent must investigate (analyze logs, inspect gradients, check architecture), diagnose the root cause, and prescribe a fix. Performance is graded on diagnosis accuracy, fix quality, investigation efficiency, and hint usage.

🏗️ Architecture

Reference-inspired layout (aligned with strong OpenEnv environments such as calendar/reasoning_gym/repl/tbench2):

torchdebug_env/
├── ARCHITECTURE.md       # Design and layering notes
├── openenv.yaml          # OpenEnv manifest
├── pyproject.toml         # Dependencies
├── models.py              # Pydantic Action/Observation/State models
├── client.py              # Typed EnvClient wrapper
├── inference.py           # Baseline LLM agent script
├── __init__.py
├── server/
│   ├── app.py             # FastAPI server entry point
│   ├── torchdebug_environment.py  # Core Environment implementation
│   └── Dockerfile         # Container build
├── scenarios/
│   ├── __init__.py        # Scenario registry
│   ├── basic_failures.py  # Task 1: Easy scenarios
│   ├── performance_issues.py  # Task 2: Medium scenarios
│   └── subtle_bugs.py     # Task 3: Hard scenarios
└── utils/
    └── reward.py          # Grading & reward computation

📋 Tasks & Scenarios

Task 1: Basic Failures (Easy) — 5 Scenarios

Task 2: Performance Issues (Medium) — 5 Scenarios

Task 3: Subtle & Compound Bugs (Hard) — 5 Scenarios

🤖 Agent Interface

Available Actions

Reward Structure

🚀 Quick Start

1. Start the Environment Server

# Install dependencies
pip install -e .

# Run locally
uvicorn server.app:app --host 0.0.0.0 --port 8000

# Or with Docker
docker build -f server/Dockerfile -t torchdebug-env .
docker run -p 8000:8000 torchdebug-env

2. Run Baseline Inference

# Required variables
export API_BASE_URL="https://api.openai.com/v1"
export MODEL_NAME="gpt-4"
export HF_TOKEN="sk-..."
python inference.py

# With HuggingFace Inference
export API_BASE_URL="https://api-inference.huggingface.co/v1"
export MODEL_NAME="meta-llama/Llama-3.3-70B-Instruct"
export HF_TOKEN="hf_..."
python inference.py

The baseline evaluates one deterministic scenario from each task (easy/medium/hard) and writes reproducible scores to outputs/evals/baseline_results.json.

3. Validate

openenv validate torchdebug_env

4. Run Pre-submission Checks (Recommended)

# Core checks (validate + local reset/step smoke tests)
python presubmit.py

# Core + deterministic grader tests are run by default
# Use --skip-tests only if your environment cannot run pytest
python presubmit.py --skip-tests

# Include container checks (docker build/run + /health + /reset)
python presubmit.py --docker

# Include baseline run (requires API_BASE_URL, MODEL_NAME, HF_TOKEN)
python presubmit.py --docker --baseline

For HF Space style external validation, use scripts/validate-submission.sh.

presubmit.py now also writes a machine-readable report to outputs/evals/submission_report.json.

📊 Baseline Results

🔧 Development

# Clone and install
git clone <repo-url>
cd torchdebug_env
pip install -e ".[dev]"

# Run tests
pytest tests/ -v

# Build Docker
openenv build torchdebug_env

🌐 HuggingFace Space Deployment

# Deploy to HuggingFace Spaces
openenv deploy torchdebug_env --space your-username/torchdebug-env

📜 License

BSD-3-Clause — Compatible with Meta/PyTorch licensing.

🏆 Hackathon Context

Built for the Meta PyTorch OpenEnv Hackathon × Scaler School (Round 1).

Why TorchDebug matters:

• —🔥 Real-world utility: Every ML engineer spends hours debugging training runs
• —🎯 Sponsor alignment: Showcases PyTorch ecosystem deeply (DDP, FSDP, AMP, transformers)
• —🧠 Progressive difficulty: Tests both basic knowledge and advanced distributed training skills
• —📈 Meaningful rewards: Partial credit for investigation — not just binary pass/fail

🧪 Judging Criteria Mapping (Round 1)

1) Real-world utility (30%)

• —Environment models a real ML engineering workflow: diagnosing failed/underperforming PyTorch training jobs.
• —Scenarios include production-like issues: device mismatch, data leakage, AMP instability, DDP/FSDP interactions.

2) Task & grader quality (25%)

• —3 difficulty tiers (easy → medium → hard) with deterministic scenario definitions.
• —Programmatic scoring in utils/reward.py returns bounded scores in $[0,1]$.
• —Grader includes anti-gaming logic and evidence-alignment incentives.

3) Environment design (20%)

• —Clean episode lifecycle via reset() / step() / state() patterns.
• —Typed action/observation/state models in models.py.
• —Reward shaping includes partial progress, efficiency terms, and hint penalties.

4) Code quality & OpenEnv compliance (15%)

• —OpenEnv manifest in openenv.yaml.
• —Local validation and smoke-check automation in presubmit.py.
• —External submission validator in scripts/validate-submission.sh.
• —Dockerized runtime via server/Dockerfile.

5) Creativity & novelty (10%)

• —Focuses on training-debug intelligence rather than benchmark gaming.
• —Hard tasks require multi-factor reasoning (numerics + systems + architecture).

🚀 Final Submission Tips (High Impact)

• —Use a hard scenario baseline in inference.py output to demonstrate non-trivial agent capability.
• —Include the generated artifacts:
• —outputs/evals/baseline_results.json
• —outputs/evals/submission_report.json
• —Before submitting HF URL, run:
• —python presubmit.py --docker --baseline
• —bash scripts/validate-submission.sh https://<your-space>.hf.space .

✅ Submission Checklist (Practical)

• —[ ] openenv validate . passes
• —[ ] python presubmit.py --docker passes
• —[ ] python presubmit.py --baseline passes with valid API credentials
• —[ ] outputs/evals/baseline_results.json is generated and committed (or attached)
• —[ ] Hugging Face Space is deployed and responds to /health and /reset

📨 Submission Portal Inputs

Use these exact links in the hackathon submission form:

• —GitHub Repository URL: https://github.com/<your-username>/<your-repo>
• —Hugging Face Space URL: https://huggingface.co/spaces/b4rty/torchdebug-env

Optional live runtime URL (for your own checks):

• —https://b4rty-torchdebug-env.hf.space

🧾 Final One-Command Validation

After setting env vars (API_BASE_URL, MODEL_NAME, HF_TOKEN), run:

python presubmit.py --docker --baseline
bash scripts/validate-submission.sh https://b4rty-torchdebug-env.hf.space .

This generates/updates:

• —outputs/evals/submission_report.json
• —outputs/evals/baseline_results.json