Datasetpublic

SWE-bench-Live/Windows

Official repository link: microsoft/SWE-bench-Live Paper link: https://arxiv.org/abs/2603.05026 Developing projects compatible on Windows platform is important to expand the user market. There are some bugs that would only occur on Windows. To migrate projects to Windows some codes need to be rewritten to have branching points across different os or use cross-platform compatible libraries... To test LLM's knowledge of Windows-specific SWE knowledge AND powershell terminal operation capability… See the full description on the dataset page: https://huggingface.co/datasets/SWE-bench-Live/Windows.

sourceHugging Facemitupdated 2d agoView on Hugging Face

0likes1.7kdownloads

Dataset Card

Official repository link: microsoft/SWE-bench-Live

Paper link: https://arxiv.org/abs/2603.05026

Developing projects compatible on Windows platform is important to expand the user market.

There are some bugs that would only occur on Windows. To migrate projects to Windows some codes need to be rewritten to have branching points across different os or use cross-platform compatible libraries...

To test LLM's knowledge of Windows-specific SWE knowledge AND powershell terminal operation capability, we propose SWE-bench-Live-Windows, a dataset of Windows specific SWE tasks.

The tasks are also native GitHub issues like SWE-bench-Live Python and MultiLang subsets.

The instances are also set up to docker images using our RepoLaunch agent (repository_link). See the "docker_image" field for image name. Note that the docker images are Windows architecture images that should be run on Windows systems.

We have successfully run our own Win-agent (https://github.com/njukenanli/Win-Agent) with different LLMs on this benchmark and posted their success rates on our leaderboard.

Dataset Schema ##

Field	Type
instance_id	str
repo	str
language	str
pull_number	str
issue_numbers	list[str]
base_commit	str
problem_statement	str
patch	str
test_patch	str
hints_text	Optional[str] from the discussions of the issue
allhintstext	Optional[str]
commit_urls	list[str]
created_at	str in ISO datetime format
commit_url	str
rebuild_cmds	list[str]
test_cmds	list[str]
log_parser	str
FAILTOPASS	list[str]
PASSTOPASS	list[str]
docker_image	str

Evaluation Protocol

[!NOTE] SWE-bench-Live evaluation strictly follows the original `SWE-bench` protocol: 1. During a rollout, the agent may access only the problem_statement field of the Hugging Face dataset and the docker image of the task instance. It must not access any other fields, such as hint, FAIL_TO_PASS, or test_patch. The test_patch must not be applied to the repository before or during the rollout. The agent must perform a single rollout based solely on the problem_statement on the docker container started from the image of the task instance. 2. Prompts, skills, and workflow instructions provided to the agent must not contain solutions specific to any task instance. They may contain only general instructions for the entire benchmark or, at most, for a specific repository. The agent must not use results from the ground-truth evaluation script to refine its solution. Compliant prompts should follow the SWE-agent prompt and the OpenHands prompt, which contain only the problem statement and general workflow instructions. When submitting results to our submissions repository, you must include your agent's raw rollout trajectories so that the maintainers can verify compliance with the SWE-bench protocol. A trajectory consists of the complete sequence of inputs to and outputs from your agent across all rollout rounds for a given task instance, including the initial prompt provided to the agent. Please follow this SWE-agent compliant trajectory example when submitting your result. If your organization's policies prohibit sharing the complete set of trajectories, you must provide at least some representative samples for verification. There is a checklist when submitting a PR to help you check whether you meet the protocol requirements again.