Datasetpublic

MERA-evaluation/SWE-MERA

SWE-MERA Continuously updated SWE-MERA dataset SWE-MERA splits: dev: for testing (10 samples) lite: presented at the leaderboard here (750 samples) full: continuously updated to collect more data (2738 samples) Load dataset from datasets import load_dataset ds = load_dataset("MERA-evaluation/SWE-MERA", split='dev') Evaluation Description The main tool to validate tasks is repotest (available at PyPI or GitHub)… See the full description on the dataset page: https://huggingface.co/datasets/MERA-evaluation/SWE-MERA.

sourceHugging Facemitupdated 8mo agoView on Hugging Face

11likes811downloads

Dataset Card

SWE-MERA

Continuously updated SWE-MERA dataset

SWE-MERA splits:

—dev: for testing (10 samples)
—lite: presented at the leaderboard here (750 samples)
—full: continuously updated to collect more data (2738 samples)

Load dataset

python

from datasets import load_dataset
ds = load_dataset("MERA-evaluation/SWE-MERA", split='dev')

Evaluation

Description

The main tool to validate tasks is repotest (available at PyPI or GitHub)

data.jsonl - dataset file, where after the agent run, the patch column was changed

Install dependencies

bash

pip install repositorytest

Run evaluation

bash

swemera --fn_input=data.jsonl --fn_output=submission.jsonl --column_patch=patch --mode=docker

If you don't want to use Docker and prefer Conda or local execution:

bash

swemera --fn_input=data.jsonl --fn_output=submission.jsonl --column_patch=patch --mode=local

Schema

Column Name	Type	Description	Is New Column
repo	VARCHAR	Repository name (e.g., GitHub repository)	False
instance_id	VARCHAR	Unique identifier for task: f"{repo.replace('/', '_') + '' + pr_number}"	False
base_commit	VARCHAR	Commit hash of the base code before patch applied	False
patch	VARCHAR	Code patch or diff applied to the base_commit to be the correct solution	False
test_patch	VARCHAR	Patch for test code, if separate from main code patch	False
problem_statement	VARCHAR	Description or statement of the problem to solve	False
hint_text	VARCHAR	Hints or tips for solving the problem from GitHub issue comments	False
created_at	VARCHAR	Date when the instance was created	False
closed_at	VARCHAR	Date when the instance was closed or resolved	True
version	VARCHAR	Version identifier for the instance or dataset	False
FAILTOPASS	LIST[VARCHAR]	Test cases which were failing but later passed	False
PASSTOPASS	LIST[VARCHAR]	Test cases which passed at all times	False
environmentsetupcommit	VARCHAR	Commit hash for environment setup configuration	False
command_build	VARCHAR	Command used to build the repo	True
command_test	VARCHAR	Command used to run the tests inside repo	True
image_name	VARCHAR	Docker or container image name used for testing environment	True
commandtestsmall	VARCHAR	Command to run a smaller subset of tests	True
timeout_build	INTEGER	Timeout limit in seconds for build command	True
timeout_test	INTEGER	Timeout limit in seconds for test command	True
meta	DICT[STR,ANY]	Additional metadata about the instance*

*meta field description:

json

{
  "score": {
    "task_correctness": "INTEGER",
    "test_correctness": "INTEGER",
    "complexity": "INTEGER",
    "test_completeness": "INTEGER"
  },
  "type": "VARCHAR",
  "pr_number": "INTEGER",
  "issue_number": "INTEGER",
  "merged_at": "VARCHAR",
  "tag": "VARCHAR // Information on how data was collected",
  "merge_commit": "VARCHAR",
  "task_id": "VARCHAR // Latency identifier of the task",
  "url": {
    "issue": "VARCHAR",
    "pr": "VARCHAR",
    "diff": "VARCHAR"
  },
  "command_test_small": "VARCHAR // Command to run only a small number of tests"
}

Dev tasks Examples

Instances

Instance ID	Short Title
reframe-0	Performance threshold goes to -inf when it should be zero.
pyflakes-1	Walrus operator + annotation can cause F821
sqlglot-2	MySQL dialect fails to parse PRIMARY KEY USING BTREE syntax
matchms-3	matchms fails when reading spectra where abundance is in scientific notation #809
guarddog-4	Add Mach-O magic bytes to bundled binary detector #523
pdoc-5	Include HTML headers in ToC
QCElemental-6	QCElemental allows floating point numbers for molecular charges, however when computing the spin with the fractional electron there is no way to represent the molecular multiplicity with just an integer.
sqlglot-7	TSQL: PRIMARY KEY constraint fails to parse if
sqlglot-8	[BUG] Conversion from SQL Server's DATEADD(quarter...) to PostgreSQL not working properly
python-markdownify-9	Inline Quotation element ignored