Mir-2002/python_code_docstring_ast_corpus

1---2task_categories:3- summarization4- text-generation5language:6- en7tags:8- code9size_categories:10- 10K<n<100K11---12 13# Overview14 15This dataset contains 34,000+ rows of code-docstring-ast data along with additional metadata. Data was gathered from various Python libraries and frameworks and their16publicly available GitHub repos. This dataset was created for the purpose of training the [CodeT5+](https://arxiv.org/abs/2305.07922) transformer on AST-enhanced code-to-doc tasks.17 18# Sources19 20The dataset was gathered from various GitHub repos sampled from [this repo by Vinta.](https://github.com/vinta/awesome-python)21 22The 26 repos are:23- matplotlib24- pytorch25- cryptography26- django27- prospector28- scikit-learn29- pandas30- numpy31- uvicorn32- feincms33- algorithms34- scrapy35- authlib36- seaborn37- coconut38- tensorflow39- flexx40- salmon41- mongo-python-driver42- virtualenv43- sphinx44- schema45- kornia46- scipy47- cherrypy48- pygame49 50Sampling was at random; I simply browsed through each category from Vinta's list and chose one from a random interesting category. 51 52# Dataset Instance53 54An instance of the dataset is as follows:55 56```57{58  <library> : <The library from which the source code came from>,59  <name> : <The name of the function/class/method>,60  <source_code> : <The raw source code itself stripped of its docstrings and comments>,61  <docstring> : <The corresponding docstring of the code>,62  <type> : <Whether it's a function, method, or class>,63  <file_path> : <The relative path of the file containing the function>,64  <ast_data> : <A flattened representation of the parsed AST. For more info about this, see section below>65}66```67 68# The AST Data69 70An extractor only focuses on specific nodes relevant for docstring generation denoted by this set:71 72```73KEEP_NODES = {74        'FunctionDef', 'AsyncFunctionDef', 'ClassDef',75        'arguments', 'arg', 'Return',76        'If', 'For', 'While', 'Try', 'With',77        'Assign', 'Call',78        'Raise', 'ExceptHandler',79        'decorator', 'bases',80        'Compare', 'BoolOp'81    }82```83 84Everything else is discarded. For example85 86**Source Code**87```88def tox_append_version_info() -> str: return '[toxfile]'89```90 91**Resulting AST Dictionary**92```93"ast_data": {94      "type": "FunctionDef",95      "children": [96        {97          "type": "arguments",98          "args": []99        },100        {101          "type": "Return",102          "has_value": true103        }104      ],105      "name": "tox_append_version_info"106    }107```108 109This dictionary is then flattened via a helper function which would then look something like `FunctionDef name:tox_append_version_info arguments Return return:yes`.110 111# Preprocessing112 113The dataset generally follows [CodeBERT's code2nl](https://github.com/microsoft/CodeBERT/tree/master/CodeBERT/code2nl) dataset cleaning standards which are as follows:114 115- Removed comments from the code116- Removed examples where code cannot be parsed into an AST117- Remove examples that documents contain special tokens (e.g. <img ...> or https:...)118- Remove examples that documents are not English119 120Furthermore, the following cleaning steps specific to this dataset were applied:121- Removed examples where, using CodeT5+'s tokenizer, the combined tokens of the source_code + ast_data is > 512122- Removed examples where, using CodeT5+'s tokenizer, the docstrings are > 512123 124# Final Statistics125 126```127{128  "original_samples": 128880,129  "processed_samples": 34537,130  "filter_stats": {131    "success": 34537,132    "non_english": 1202,133    "docstring_too_long": 3848,134    "input_too_long": 8366,135    "docstring_too_short": 74013,136    "error": 0,137    "error: unhashable type: 'list'": 6914138  },139  "split_sizes": {140    "train": 24175,141    "val": 5180,142    "test": 5182143  },144  "input_token_stats": {145    "min": 16,146    "max": 505,147    "avg": 164.071148  },149  "target_token_stats": {150    "min": 4,151    "max": 254,152    "avg": 52.758153  },154  "type_distribution": {155    "method": 12682,156    "function": 8733,157    "class": 2760158  }159}160 161```162 163# NOTE164 165This dataset is *imperfect*. Use under your own volition.