FineEnvs/repo2rlenv-cve-patches


repo2rlenv-cve-patches

Generated by **Repo2RLEnv** — turning real GitHub repositories into verifiable RL environments.

💡 Browse this dataset in your browser — click the badge above or open `HuggingFaceH4/harbor-visualiser` to inspect every task's spec, instruction, oracle patch, test script, and Dockerfile.

• —Source repos (6):
• —`Pylons/waitress`
• —`andialbrecht/sqlparse`
• —`lepture/mistune`
• —`pallets/flask`
• —`pallets/werkzeug`
• —`psf/requests`
• —Pipeline: `cve_patches`
• —Tasks: 19
• —Visibility: public
• —Spec: Harbor task format with the [metadata.repo2env] extension

How it was generated

Each task in this dataset was produced by the `cve_patches` pipeline. The pipeline mines real merged pull requests / commits from the source repo(s), applies quality filters, strips information-leakage from the instruction text, and emits a Harbor-shaped task directory with the gold patch as the oracle.

Reproduce locally:

pip install repo2rlenv
repo2rlenv generate \
  --repo <owner>/<repo> \
  --pipeline cve_patches \
  --pipeline-opt limit=10 \
  --out ./datasets/my-cve_patches

See the pipeline docs for the full option list + reward design.

Run with Harbor

Each task ships a environment/Dockerfile and tests/test.sh, so you can score patches end-to-end:

# Pull the dataset locally
repo2rlenv pull AdithyaSK/repo2rlenv-cve-patches /tmp/repo2rlenv-cve-patches

# Confirm structural soundness — oracle adapter applies the gold patch
# and must score reward = 1.000
harbor run -p /tmp/repo2rlenv-cve-patches -a oracle --env docker

# Score an agent (claude-code + Sonnet 4.6)
harbor run \
  -p /tmp/repo2rlenv-cve-patches \
  -a claude-code -m anthropic/claude-sonnet-4-6 \
  --ak max_budget_usd=2.00 \
  --ae ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
  --env docker

The reward is test-execution (graded F2P/P2P). After your patch is applied, tests/test.sh runs the suite and a baked verifier scores reward = f2p_rate × p2p_rate to /logs/verifier/reward.txt (a dense training signal), and writes the strict SWE-bench resolved bool plus a breakdown to /logs/verifier/reward-details.json:

{"reward": 1.0, "resolved": true, "f2p_passed": 3, "f2p_total": 3,
 "p2p_passed": 595, "p2p_total": 595, "regressions": [], "parse_status": "ok"}

resolved requires all FAILTOPASS to pass AND all PASSTOPASS to be maintained. No API key is needed — grading is purely test-based.

Reward signal

The reward function is part of the task itself (tests/test.sh + the verifier code baked into the image). The full per-task breakdown is written to /logs/verifier/reward-details.json at run time — useful for slicing training data by component.

See the pipeline doc for the component-by-component design.

Layout

tasks/
└── <task-id>/
    ├── task.toml          # Harbor task with [metadata.repo2env]
    ├── instruction.md     # natural-language prompt
    ├── solution/
    │   ├── patch.diff     # oracle (gold) diff
    │   └── solve.sh       # oracle adapter applies patch.diff
    ├── environment/
    │   └── Dockerfile     # builds the task's container
    └── tests/
        └── test.sh        # verifier — writes /logs/verifier/reward.txt

License

Apache-2.0 — same as Repo2RLEnv itself. The original PR contents remain under their respective source-repo licenses; this dataset redistributes public commits under fair-use for ML research / training-data purposes.

Citation

@misc{fineenvs,
  author = {Kolavi, Adithya S},
  title  = {FineEnvs: Open Source RL Environments for LLM Agents},
  year   = {2026},
  url    = {https://github.com/adithya-s-k/FineEnvs}
}