OpenMOSS-Team/SWE-bench-Science

1---2pretty_name: SWE-bench Science3language:4  - en5tags:6  - code7  - software-engineering8  - scientific-computing9  - coding-agents10  - benchmark11  - long-horizon12  - harbor13  - pier14size_categories:15  - n<1K16license: mit17configs:18  - config_name: default19    data_files:20      - split: test21        path: data/tasks.csv22    features:23      - name: task_id24        dtype: string25      - name: science_knowledge_ablation26        dtype: bool27      - name: title28        dtype: string29      - name: domain30        dtype: string31      - name: language32        dtype: string33      - name: repository_url34        dtype: string35      - name: base_commit36        dtype: string37      - name: source_license38        dtype: string39      - name: gpl_family40        dtype: bool41      - name: restricted_license42        dtype: bool43      - name: license_gate44        dtype: string45      - name: material_license46        dtype: string47      - name: material_license_source48        dtype: string49      - name: material_restricted50        dtype: string51      - name: materials_gate52        dtype: string53      - name: materials_manifest_sha25654        dtype: string55      - name: material_licenses56        dtype: string57      - name: materials_provenance58        dtype: string59      - name: restricted_reason60        dtype: string61      - name: environment_image62        dtype: string63      - name: verifier_image64        dtype: string65      - name: image_platform66        dtype: string67      - name: task_path68        dtype: string69      - name: status70        dtype: string71---72 73# SWE-bench Science74 75SWE-bench Science evaluates coding agents on software-engineering tasks drawn from scientific-computing repositories. The release contains 119 tasks across 20 scientific domains, with isolated environments and separate programmatic verifiers.76 77- **GitHub release repository:** [OpenMOSS/SWE-bench-Science](https://github.com/OpenMOSS/SWE-bench-Science)78- **Runtime images:** [Docker Hub](https://hub.docker.com/u/kevinxulearning), pinned by immutable `linux/amd64` digests79- **Evaluation framework:** [Pier](https://github.com/datacurve-ai/pier), compatible with Harbor task format80 81## Dataset Summary82 83| Metric | Value |84| --- | ---: |85| Tasks | 119 |86| Scientific domains | 20 |87| Default selection | 96 unrestricted-license tasks |88| Restricted selection | 23 tasks |89| GPL/LGPL/AGPL-family tasks | 18 |90| Environment images | 119 Docker Hub images |91| Verifier images | 119 Docker Hub images |92| Image platform | `linux/amd64` |93 94The `science_knowledge_ablation` column is `true` for the 91-task science-knowledge95split used by the ablation experiment. Its release IDs are `002-082`, `084`, `086`,96`090`, `097-101`, `111`, and `114`; all other rows are `false`.97 98## Dataset Viewer And Files99 100The Dataset Viewer reads the canonical [`data/tasks.csv`](data/tasks.csv) table and generates its preview automatically. The release does not commit a duplicate Parquet export, so the CSV remains the single source of truth for the 119 task rows.101 102The repository also includes:103 104| Path | Purpose |105| --- | --- |106| `data/tasks.csv` | Human-readable task table |107| `data/statistics.md` | Generated release statistics |108| `manifests/tasks.jsonl` | Canonical machine-readable release manifest |109| `manifests/release_provenance.json` | Authoritative configuration and clean-verification provenance |110| `selections/` | Reproducible task selections |111| `tasks/task_NNN/` | Thin Harbor/Pier task bundles |112| `scripts/` | Materialization, provider, batch, and summary tools |113| `docs/run-batch.md` | Full provider and batch-runner reference |114 115The environment image contains the baseline source, public fixtures, dependencies, and compilers. The separate verifier image contains held-out tests and the grader. The dataset does not contain reference-answer patches, credentials, agent trajectories, or private verifier tests.116 117## Quick Start118 119The pinned `datacurve-pier==0.3.0` release requires Python 3.12 or newer.120 121~~~bash122python3.12 -m pip install "huggingface_hub[cli]"123hf auth login124hf download OpenMOSS-Team/SWE-bench-Science \125  --repo-type dataset --local-dir swe-bench-science126cd swe-bench-science127 128uv tool install --python 3.12 "datacurve-pier==0.3.0"129docker login130~~~131 132Materialize the default selection:133 134~~~bash135python3 scripts/materialize.py \136  --output tasks-selected --force137~~~138 139Materialize one task, a comma-separated list, or inclusive ranges:140 141~~~bash142python3 scripts/materialize.py \143  --task-id 002,005-007 \144  --output tasks-selected-small --force145~~~146 147Materialize the complete 91-task science-knowledge ablation split. It contains148restricted-license tasks, so the explicit license opt-in is required:149 150~~~bash151python3 scripts/materialize.py \152  --task-id 002-082,084,086,090,097-101,111,114 \153  --allow-restricted-licenses \154  --output tasks-science-knowledge-ablation --force155~~~156 157Every materialization writes `selection.json` with the exact task IDs used for the run.158 159## Restricted Licenses160 161Twenty-three tasks are excluded from the default selection because they contain GPL/LGPL/AGPL-family code, academic non-commercial sources or materials, or restricted third-party data. Include them only after confirming that your use is permitted:162 163~~~bash164python3 scripts/materialize.py \165  --allow-restricted-licenses \166  --output tasks-selected-all --force167~~~168 169The GPL/LGPL/AGPL-family task IDs are `003, 020, 021, 023, 032, 057, 066, 074, 075, 082, 083, 084, 085, 096, 097, 098, 100, 118`. Tasks `019`, `026`, `035`, `101`, and `102` are restricted for other reasons. There is no `--allow-GPL` option. The selection flag controls which task bundles are materialized; it does not replace the upstream license obligations.170 171## Run An Evaluation172 173Run an infrastructure smoke with no model:174 175~~~bash176pier run -p tasks-selected-small \177  --agent nop --env docker \178  --n-concurrent 1 --n-attempts 1 \179  --no-force-build --no-delete --yes180~~~181 182Run a real agent by selecting a harness, model, and provider profile:183 184~~~bash185# Claude Code186pier run -p tasks-selected-small \187  --agent claude-code --env docker \188  --env-file ~/.config/swe-bench-science/claude.env \189  --model anthropic/claude-opus-4-7 --n-concurrent 1190 191# mini-swe-agent192pier run -p tasks-selected-small \193  --agent mini-swe-agent --env docker \194  --env-file ~/.config/swe-bench-science/mini-swe-agent.env \195  --model openai/gpt-5 --n-concurrent 1196~~~197 198For Codex gateway profiles, use the included wrapper:199 200~~~bash201python3 scripts/run_batch.py \202  --path tasks-selected-small \203  --agent codex \204  --env-file ~/.config/swe-bench-science/codex.env \205  --n-concurrent 2 --n-attempts 1 \206  --jobs-dir jobs --job-name codex-small207~~~208 209For an approximately 120-second agent-stage smoke, add210`--agent-timeout-multiplier 0.0223`; for an approximately 30-second smoke, use211`--agent-timeout-multiplier 0.0055556`. Verifier and scientific-build timeouts212remain independent. The pinned `datacurve-pier==0.3.0` release requires Python2133.12 or newer.214 215## Provider Profiles216 217Create profiles outside the downloaded dataset:218 219~~~bash220mkdir -p ~/.config/swe-bench-science221cp profiles/codex.env.example ~/.config/swe-bench-science/codex.env222cp profiles/claude.env.example ~/.config/swe-bench-science/claude.env223cp profiles/mini-swe-agent.env.example ~/.config/swe-bench-science/mini-swe-agent.env224chmod 600 ~/.config/swe-bench-science/*.env225~~~226 227Codex profiles use `MODEL`, `OPENAI_API_KEY`, `CODEX_BASE_URL`, `CODEX_WIRE_API`, `CODEX_VERSION`, and `CODEX_REASONING_EFFORT`. Set `CODEX_WIRE_API=responses` for the OpenAI Responses API or `chat` for Chat Completions. Claude Code uses `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_BASE_URL`, and optional `ANTHROPIC_CUSTOM_HEADERS`. mini-swe-agent uses the provider variables expected by its selected model adapter.228 229Credentials are read at runtime. They are not stored in task metadata, Dockerfiles, image layers, or result summaries.230 231## Results232 233Pier writes one aggregate result and one trial directory per task and attempt:234 235~~~text236jobs/<job-name>/result.json237jobs/<job-name>/<task>__<trial>/verifier/reward.json238jobs/<job-name>/<task>__<trial>/verifier/ctrf.json239jobs/<job-name>/<task>__<trial>/verifier/test-stdout.txt240~~~241 242The wrapper additionally writes `jobs/summary.json` and `jobs/summary.csv`. For a direct Pier run, generate the same summaries with:243 244~~~bash245python3 scripts/summarize_results.py --jobs-dir jobs246~~~247 248Use `pier view jobs` to inspect trajectories.249 250See [`docs/run-batch.md`](docs/run-batch.md) for the complete option reference,251gateway/profile configuration, dry-run mode, retry and timeout controls, and252result paths.253 254## Licensing And Attribution255 256The dataset card, release metadata, and helper tools use the repository's MIT terms. Task source, papers, figures, fixtures, and other third-party materials retain their upstream licenses. The source-license field does not automatically license copied scientific materials; audited material notices and modification notes are retained in the relevant task bundles.257 258The dataset is independent of GitHub at runtime. After download, materialization and evaluation use the local task bundle and the Docker Hub image digests recorded in `task.toml` and `data/tasks.csv`.259