anon-iclr-submission/benchname-project-level-code-completion

🥷 BenchName (Project-level code completion)

This is the benchmark for Project-level code completion task as part of the 🥷 BenchName benchmark. Each datapoint contains the file for completion, a list of lines to complete with their categories (see the categorization below), and a repository snapshot that can be used to build the context. All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). The datapoints can be removed upon request.

How-to

Load the data via load_dataset:

from datasets import load_dataset

config_names = [
  'small_context',
  'medium_context',
  'large_context',
  'huge_context'
]

ds = load_dataset('anon-iclr-submission/benchname-project-level-code-completion', config_name, split='test')

Dataset Structure

Datapoints in the dataset have the following structure:

• —repo – repository name in the format {GitHub_user_name}__{repository_name}
• —commit_hash – commit hash of the repository
• —completion_file – dictionary with the completion file content in the following format:
• —filename – path to the completion file
• —content – content of the completion file
• —completion_lines – dictionary where the keys are categories of lines and values are a list of integers (numbers of lines to complete). The categories are:
• —committed – line contains at least one function or class from the files that were added on the completion file commit
• —inproject – line contains at least one function or class from the repository snapshot at the moment of completion
• —infile – line contains at least one function or class from the completion file
• —common – line contains at least one function or class with common names, e.g., main, get, etc.
• —non_informative – line that was classified to be non-informative, e.g., too short, contains comments, etc.
• —random – other lines.
• —repo_snapshot – dictionary with a snapshot of the repository before the commit. It has the same structure as completion_file, but filenames and contents are orginized as lists.
• —completion_lines_raw – same as completion_lines, but before sampling

How we collected the data

To collect the data, we cloned repositories from GitHub where the main language is Python. The completion file for each datapoint is a .py file that was added to the repository in a commit. The state of the repository before this commit is the repo snapshot.

The dataset configurations are based on the number of characters in .py files from the repository snapshot:

• —small_context – less than 48K characters;
• —medium_context – from 48K to 192K characters;
• —large_context – from 192K to 768K characters;
• —huge_context – more than 768K characters.

Datasets Stats

Completion File

Repository Snapshot .py files

Repository Snapshot non .py files

Line Counts:

Scores

You can find the results of running various models on this dataset in our leaderboard.