Datasetpublic

litble/Multi-Docker-Eval

Dataset Summary Multi-Docker-Eval is a multi-language, multi-dimensional benchmark designed to rigorously evaluate the capability of Large Language Model (LLM)-based agents in automating a critical yet underexplored task: constructing executable Docker environments for real-world software repositories. How to Use from datasets import load_dataset ds = load_dataset('litble/Multi-Docker-Eval') Dataset Structure The data format of Multi-Docker-Eval is… See the full description on the dataset page: https://huggingface.co/datasets/litble/Multi-Docker-Eval.

sourceHugging Faceapache-2.0updated 10mo agoView on Hugging Face

1likes62downloads

Dataset Card

Dataset Summary

Multi-Docker-Eval is a multi-language, multi-dimensional benchmark designed to rigorously evaluate the capability of Large Language Model (LLM)-based agents in automating a critical yet underexplored task: constructing executable Docker environments for real-world software repositories.

How to Use

python

from datasets import load_dataset
ds = load_dataset('litble/Multi-Docker-Eval')

Dataset Structure

The data format of Multi-Docker-Eval is directly compatible with the swe-factory and RepoLaunch frameworks.

Field name	Type	Description
`instance_id`	str	A formatted instance identifier, usually as `repo_owner__repo_name-PR-number`.
`repo`	str	The repository owner/name identifier from GitHub.
`pull_number`	int64	The GitHub pull request number that contains the solution (patch and test_patch).
`issue_numbers`	list(str)	A list of the GitHub issue numbers that are addressed and resolved by this pull request.
`base_commit`	str	The commit hash of the repository representing the HEAD of the repository before the solution PR is applied.
`patch`	str	The gold patch, the patch generated by the PR (minus test-related code), that resolved the issue.
`test_patch`	str	A test-file patch that was contributed by the solution PR.
`problem_statement`	str	The issue title and body.
`hints_text`	str	Comments made on the issue prior to the creation of the solution PR’s first commit creation date.
`created_at`	timestamp[s]	The creation date of the pull request.
`language`	str	The programming language of the repository.
`label`	str	Classification of environmental configuration difficulty (Easy/Hard).