CoolFace
Datasetpublic

anon-iclr-submission/benchname-module-summarization

🥷 BenchName (Module summarization) This is the benchmark for Module summarization task as part of the 🥷 BenchName benchmark. The current version includes 216 manually curated text files describing different documentation of open-source permissive Python projects. The model is required to generate such description, given the relevant context code and the intent behind the documentation. All the repositories are published under permissive licenses (MIT, Apache-2.0… See the full description on the dataset page: https://huggingface.co/datasets/anon-iclr-submission/benchname-module-summarization.

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
0likes16downloads
Dataset Card

🥷 BenchName (Module summarization)

This is the benchmark for Module summarization task as part of the 🥷 BenchName benchmark. The current version includes 216 manually curated text files describing different documentation of open-source permissive Python projects. The model is required to generate such description, given the relevant context code and the intent behind the documentation. All the repositories are published under permissive licenses (MIT, Apache-2.0, BSD-3-Clause, and BSD-2-Clause). The datapoints can be removed upon request.

How-to

Load the data via `load_dataset`:

    from datasets import load_dataset

    dataset = load_dataset("anon-iclr-submission/benchname-module-summarization")

Datapoint Structure

Each example has the following fields:

**Field****Description**
repoName of the repository
target_textText of the target documentation file
docfile_nameName of the file with target documentation
intentOne sentence description of what is expected in the documentation
licenseLicense of the target repository
relevant_code_filesPaths to relevant code files (files that are mentioned in target documentation)
relevant_code_dirPaths to relevant code directories (directories that are mentioned in target documentation)
path_to_docfilePath to file with documentation (path to the documentation file in source repository)
relevant_code_contextRelevant code context collected from relevant code files and directories

Note: you may collect and use your own relevant context. Our context may not be suitable. Zipped repositories can be found the repos directory.

Metric

To compare the predicted documentation and the ground truth documentation, we introduce the new metric based on LLM as an assessor. Our approach involves feeding the LLM with relevant code and two versions of documentation: the ground truth and the model-generated text. The LLM evaluates which documentation better explains and fits the code. To mitigate variance and potential ordering effects in model responses, we calculate the probability that the generated documentation is superior by averaging the results of two queries with the different order.