CoolFace
Datasetpublic

JetBrains-Research/lca-module-summarization

🏟️ Long Code Arena (Module summarization) This is the benchmark for Module summarization task as part of the 🏟️ Long Code Arena benchmark. The current version includes 216 manually curated text files describing different documentation of open-source permissive Python projects. The model is required to generate such description, given the relevant context code and the intent behind the documentation. All the repositories are published under permissive licenses (MIT… See the full description on the dataset page: https://huggingface.co/datasets/JetBrains-Research/lca-module-summarization.

sourceHugging Faceapache-2.0updated 2y agoView on Hugging Face
1likes86downloads
Dataset Card

🏟️ Long Code Arena (Module summarization)

This is the benchmark for Module summarization task as part of the 🏟️ Long Code Arena benchmark. The current version includes 216 manually curated text files describing different documentation of open-source permissive Python projects. The model is required to generate such description, given the relevant context code and the intent behind the documentation. All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). The datapoints can be removed upon request.

How-to

Load the data via `load_dataset`:

    from datasets import load_dataset

    dataset = load_dataset("JetBrains-Research/lca-module-summarization")

Datapoint Structure

Each example has the following fields:

**Field****Description**
repoName of the repository
target_textText of the target documentation file
docfile_nameName of the file with target documentation
intentOne sentence description of what is expected in the documentation
licenseLicense of the target repository
relevant_code_filesPaths to relevant code files (files that are mentioned in target documentation)
relevant_code_dirPaths to relevant code directories (directories that are mentioned in target documentation)
path_to_docfilePath to file with documentation (path to the documentation file in source repository)
relevant_code_contextRelevant code context collected from relevant code files and directories

Note: you may collect and use your own relevant context. Our context may not be suitable. Zipped repositories can be found the repos directory.

Metric

To compare the predicted documentation and the ground truth documentation, we introduce the new metric based on LLM as an assessor. Our approach involves feeding the LLM with relevant code and two versions of documentation: the ground truth and the model-generated text. The LLM evaluates which documentation better explains and fits the code. To mitigate variance and potential ordering effects in model responses, we calculate the probability that the generated documentation is superior by averaging the results of two queries with the different order.

For more details about metric implementation, please refer to our GitHub repository.

Citing

@article{bogomolov2024long,
  title={Long Code Arena: a Set of Benchmarks for Long-Context Code Models},
  author={Bogomolov, Egor and Eliseeva, Aleksandra and Galimzyanov, Timur and Glukhov, Evgeniy and Shapkin, Anton and Tigina, Maria and Golubev, Yaroslav and Kovrigin, Alexander and van Deursen, Arie and Izadi, Maliheh and Bryksin, Timofey},
  journal={arXiv preprint arXiv:2406.11612},
  year={2024}
}

You can find the paper here.