Redhanuman/sabnock-docker

Sabnock Docker

Sabnock Docker is a synthetic benchmark for testing whether AI agents can repair real Docker, Compose, and containerized CI failures.

It is built for AI engineers, not model hype. Each task contains a broken mini-repository, failing build/runtime logs, an expected fix, and machine-checkable scoring hints.

Why This Exists

Most coding-agent demos look good until the work touches Docker, Compose, dependency resolution, build context, network binding, secrets, or CI logs. Sabnock Docker targets that exact failure class.

The v1 benchmark contains 108 synthetic tasks covering:

• —Missing build-context files
• —npm ci lockfile failures
• —Multi-stage binary copy mistakes
• —Compose readiness bugs
• —.dockerignore production artifact mistakes
• —Alpine native dependency failures
• —Python src/ layout container bugs
• —Poetry --no-root mistakes
• —Compose bind mounts hiding node_modules
• —BuildKit secret handling
• —apt-get update ordering
• —Container network binding failures

Repo Layout

data/sabnock_docker_seed.json   Hand-written seed tasks
data/sabnock_docker_v1.json     108-task benchmark dataset
data/sabnock_docker_*.json      HF-ready split files
sabnock_docker/                 Loader, validator, scorer, CLI
scripts/build_dataset.py        Deterministic v1 dataset builder
app.py                          Hugging Face Space UI
tests/                          Dataset and scorer smoke tests

Dataset Row Shape

Each row has:

{
  "id": "sbd-0001",
  "split": "seed",
  "title": "Python image forgets requirements.txt",
  "failure_type": "missing_build_context_file",
  "ecosystem": "python",
  "difficulty": "easy",
  "files": [{"path": "Dockerfile", "content": ["FROM python:3.12-slim"]}],
  "failure_log": ["ERROR: Could not open requirements file"],
  "build": {"command": "docker build -t sabnock/sbd-0001 ."},
  "verify": {"commands": ["docker build -t sabnock/sbd-0001 ."]},
  "oracle_patch": ["diff --git a/Dockerfile b/Dockerfile"],
  "scoring": {
    "must_touch": ["Dockerfile"],
    "must_contain": [{"path": "Dockerfile", "text": "COPY requirements.txt ."}],
    "must_not_contain": []
  }
}

The content, failure_log, and oracle_patch fields are arrays of lines so the dataset stays readable on Hugging Face.

Splits

seed        12 hand-written tasks
train       72 generated training/evaluation tasks
validation  12 validation tasks
challenge   12 harder public challenge tasks

The public challenge split is a placeholder for leaderboard-style scoring. For a serious leaderboard, keep future hidden tasks private and run them in a controlled evaluator.

Local Use

Rebuild the v1 dataset:

python3 scripts/build_dataset.py

Validate the dataset:

python3 -m sabnock_docker.cli validate

Print benchmark statistics:

python3 -m sabnock_docker.cli stats

List tasks:

python3 -m sabnock_docker.cli list

Filter tasks:

python3 -m sabnock_docker.cli list --ecosystem compose --difficulty medium

Export a broken task to a directory:

python3 -m sabnock_docker.cli export sbd-0001 /tmp/sbd-0001

Score a proposed patch statically:

python3 -m sabnock_docker.cli score-patch sbd-0001 fix.patch

Verify with Docker locally:

python3 -m sabnock_docker.cli verify sbd-0001 --patch fix.patch

The score-patch command applies the patch to a temp copy of the broken repo and checks the final files. The verify command exports the task, applies the patch with git apply, then runs the task's verification commands. It requires Docker for Docker-backed checks.

Render a Markdown report:

python3 -m sabnock_docker.cli report -o BENCHMARK.md

Score a JSONL agent submission file:

python3 -m sabnock_docker.cli score-submissions submissions.jsonl

See SUBMISSIONS.md and schemas/submission.schema.json for the leaderboard submission format.

Hugging Face Space

Run the browser UI:

pip install -r requirements.txt
python3 app.py

The Space lets people browse tasks, filter the benchmark, inspect files/logs, score a patch, and view a leaderboard scaffold. Docker execution is intentionally local-first because public hosted Spaces should not run arbitrary Docker builds.

Leaderboard Metrics

Recommended metrics:

• —pass_at_1: first-patch success rate
• —patch_apply_rate: whether the patch applies cleanly
• —docker_pass_rate: whether local Docker verification passes
• —median_runtime_seconds: wall time per task
• —mean_patch_lines: patch size
• —cost_usd: approximate model/tool cost

Submission row:

{"agent":"my-agent","task_id":"sbd-0001","patch":"diff --git ...","runtime_seconds":12.4,"cost_usd":0.03}

License

Apache-2.0. All seed tasks are synthetic and original.