semeru/Text-Code-CodeSearchNet-Python

1---2license: mit3Programminglanguage: "Python"4version: "N/A"5Date: "Codesearchnet(Jun 2020 - paper release date)"6Contaminated: "Very Likely"7Size: "Standard Tokenizer (TreeSitter)"8---9 10### Dataset is imported from CodeXGLUE and pre-processed using their script.11 12# Where to find in Semeru:13The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/text-to-code/codesearchnet/python in Semeru14 15# CodeXGLUE -- Code Search (AdvTest)16 17## Task Definition18 19Given a natural language, the task is to search source code that matches the natural language. To test the generalization ability of a model,  function names and variables in test sets are replaced by special tokens.20 21## Dataset22 23The dataset we use comes from [CodeSearchNet](https://arxiv.org/pdf/1909.09436.pdf) and we filter the dataset as the following:24 25- Remove examples that codes cannot be parsed into an abstract syntax tree.26- Remove examples that #tokens of documents is < 3 or >25627- Remove examples that documents contain special tokens (e.g. <img ...> or https:...)28- Remove examples that documents are not English.29 30Besides, to test the generalization ability of a model,  function names and variables in test sets are replaced by special tokens.31 32 33### Data Format34 35After preprocessing dataset, you can obtain three .jsonl files, i.e. train.jsonl, valid.jsonl, test.jsonl36 37For each file, each line in the uncompressed file represents one function.  One row is illustrated below.38 39  - **repo:** the owner/repo40  - **path:** the full path to the original file41  - **func_name:** the function or method name42  - **original_string:** the raw string before tokenization or parsing43  - **language:** the programming language44  - **code/function:** the part of the `original_string` that is code45  - **code_tokens/function_tokens:** tokenized version of `code`46  - **docstring:** the top-level comment or docstring, if it exists in the original string47  - **docstring_tokens:** tokenized version of `docstring`48  - **url:** the url for the example (identify natural language)49  - **idx**: the index of code (identify code)50 51### Data Statistics52 53Data statistics of the dataset are shown in the below table:54 55|       | #Examples |56| ----- | :-------: |57| Train |  251,820  |58| Dev   |   9,604   |59| Test  |  19,210   |60 61 62### Example63 64Given a text-code file evaluator/test.jsonl:65 66```json67{"url": "url0", "docstring": "doc0","function": "fun0", "idx": 10}68{"url": "url1", "docstring": "doc1","function": "fun1", "idx": 11}69{"url": "url2", "docstring": "doc2","function": "fun2", "idx": 12}70{"url": "url3", "docstring": "doc3","function": "fun3", "idx": 13}71{"url": "url4", "docstring": "doc4","function": "fun4", "idx": 14}72```73 74 75### Input Predictions76 77For each url for natural language, descending sort candidate codes and return their idx in order. For example:78 79```json80{"url": "url0", "answers": [10,11,12,13,14]}81{"url": "url1", "answers": [10,12,11,13,14]}82{"url": "url2", "answers": [13,11,12,10,14]}83{"url": "url3", "answers": [10,14,12,13,11]}84{"url": "url4", "answers": [10,11,12,13,14]}85```86 87 88## Reference89<pre><code>@article{husain2019codesearchnet,90  title={Codesearchnet challenge: Evaluating the state of semantic code search},91  author={Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},92  journal={arXiv preprint arXiv:1909.09436},93  year={2019}94}</code></pre>