icmlbenchname/bug-localization

BenchName (Bug localization)

This is the benchmark for the Bug localization task as part of the BenchName benchmark.

The bug localization problem can be formulated as follows: given an issue with a bug description and a repository snapshot in a state where the bug is reproducible, identify the files within the repository that need to be modified to address the reported bug.

The dataset provides all the required components for evaluation of bug localization approaches in real project-level large-scale data collected from GitHub, including:

• —Bug issue description;
• —Repositories, from which the content can be extracted at the state of the commit SHA where the bug is reproducible;
• —List of files that should be changed in order to solve the bug;
• —Other additional data and metrics that can be useful in developing new approaches.

All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). The datapoints can be removed upon request.

The collected dataset was carefully filtered, enhanced with useful metrics and, what's more, manually labeled, which assures the data quality and provides a golden subset of good examples for evaluation.\ Moreover, the dataset was split into several categories, namely:

...and splits, namely: | Split | Description | |:------------------:|:----------------------------------------:| | dev | All collected datapoints | | test | Manually verified datapoints | | train | Rest of the datapoint from dev without test |

The results of evaluation of various bug localization approaches can be found in the BenchName leaderboard.

The following sections describe the utilities around the dataset, as well as dataset content.

How-to

• —Load the data via `load_dataset`:

    from datasets import load_dataset
    
    # Select a configuration from ["py", "java", "kt"]
    configuration = "py"
    # Select a split from ["dev", "train", "test"]
    split = "dev"
    # Load data
    dataset = load_dataset("icmlbenchname/bug-localization", configuration, split=split)  

• —Load repos via `hf_hub_download`:

    from huggingface_hub import hf_hub_download
    from datasets import load_dataset
    import zipfile
    import os
    
    # Load json with list of repos' .tar.gz file paths
    paths_json = load_dataset("icmlbenchname/bug-localization", data_files="repos.json")

    # Load each repo in .tar.gz format, unzip, delete archive
    repos = paths_json[configuration][0]
    repos_path = "path/to/repos"

    for i, repo_zip_path in enumerate(repos):
        repo_name = os.path.basename(repo_zip_path).split('.')[0]
        repo_path = os.path.join(repos_path, repo_name)

        # Load repo zip
        local_repo_zip_path = hf_hub_download(
            "icmlbenchname/bug-localization",
            filename=repo_zip_path,
            repo_type="dataset",
            local_dir="path/to/zips"
        )

        # Unzip repo
        with zipfile.ZipFile(local_repo_zip_path, 'r') as zip_ref:
            zip_ref.extractall(repo_path)
        os.remove(local_repo_zip_path)

• —Data streaming via HFDataSource. Besides data loading, HFDataSource returns a datapoint for running the baseline along with the content of the repository at the state where the bug is reproducible (aka base_sha commit of the pull request that resloves the bug issue). \ All source code for working with the Git history of repositories (commits, diffs, etc.) is available in `git_utils.py`, as is an example of baselines, utilizing this dataset.

Bug localization data

Each datapoint contains the main fields, as well as additional metrics calculated on them. \ The main fields are:

\* Excluding test files that do not contain bug causes, rather changed in order to add tests for proving that the bug is gone.

The metrics-related fields are: | Field | Description | |:------------------:|:----------------------------------------:| | changed_files_exts | Dictionary from the extension of changed files to their count. | | changed_files_count | Number of changed files. | | java_changed_files_count | Number of changed .java files. | | kt_changed_files_count | Number of changed .kt files. | | py_changed_files_count | Number of changed .py files. | | code_changed_files_count | Number of changed .java, .kt, or .py files. | | repo_symbols_count | Number of symbols in the repository. | | `repo_tokens_count` | Number of tokens* in the files of the repository. | | `repo_lines_count` | Number of lines in the files of the repository. | | repo_files_without_tests_count | Number of files in the repository. | | `changed_symbols_count` | Number of symbols in the changed lines in diff. | | changed_tokens_count | Number of tokens in the changed lines in diff. | | `changed_lines_count` | Number of changed lines in diff (including added and deleted). | | changed_files_without_tests_count | Number of files in diff. | | `issue_symbols_count` | Number of symbols in the issue body. | | issue_words_count | Number of words in the issue body (separated by space symbols). | | `issue_tokens_count` | Number of tokens* in the issue body. | | `issue_lines_count` | Number of text lines in the issue body (separated by \\n). | | issue_links_count | Number of links (\...\) present in the issue body. | | issue_code_blocks_count | Number of code blocks (\\\...\\\) present in the issue body. |

\ Excluding test files that do not contain bug causes, rather changed in order to add tests for proving that the bug is gone. \ \\* Using GPT-4 tokenizer via ticktoken.