semeru/code-code-CodeCompletion-TokenLevel-Java

1---2license: mit3Programminglanguage: "Java"4version: "N/A"5Date: "From paper:  https://homepages.inf.ed.ac.uk/csutton/publications/msr2013.pdf (2013 - paper release date)"6Contaminated: "Very Likely"7Size: "Standard Tokenizer (TreeSitter)"8---9 10### Dataset is imported from CodeXGLUE and pre-processed using their script.11 12# Where to find in Semeru:13The dataset can be found at /nfs/semeru/semeru_datasets/code_xglue/code-to-code/CodeCompletion-token/dataset/javaCorpus in Semeru14 15 16# CodeXGLUE -- Code Completion (token level)17 18**Update 2021.07.30:** We update the code completion dataset with literals normalized to avoid sensitive information. 19 20Here is the introduction and pipeline for token level code completion task.21 22## Task Definition23 24Predict next code token given context of previous tokens. Models are evaluated by token level accuracy.25 26Code completion is a one of the most widely used features in software development through IDEs. An effective code completion tool could improve software developers' productivity. We provide code completion evaluation tasks in two granularities -- token level and line level. Here we introduce token level code completion. Token level task is analogous to language modeling. Models should have be able to predict the next token in arbitary types.27 28 29## Dataset30 31The dataset is in java.32 33 34### Dependency35 36- javalang == 0.13.037 38 39 40 41### Github Java Corpus42 43We use java corpus dataset mined by Allamanis and Sutton, in their MSR 2013 paper [Mining Source Code Repositories at Massive Scale using Language Modeling](https://homepages.inf.ed.ac.uk/csutton/publications/msr2013.pdf). We follow the same split and preprocessing in Karampatsis's ICSE 2020 paper [Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code](http://homepages.inf.ed.ac.uk/s1467463/documents/icse20-main-1325.pdf).44 45 46 47### Data Format48 49Code corpus are saved in txt format files. one line is a tokenized code snippets:50```51<s> from __future__ import unicode_literals <EOL> from django . db import models , migrations <EOL> class Migration ( migrations . Migration ) : <EOL> dependencies = [ <EOL> ] <EOL> operations = [ <EOL> migrations . CreateModel ( <EOL> name = '<STR_LIT>' , <EOL> fields = [ <EOL> ( '<STR_LIT:id>' , models . AutoField ( verbose_name = '<STR_LIT>' , serialize = False , auto_created = True , primary_key = True ) ) , <EOL> ( '<STR_LIT:name>' , models . CharField ( help_text = b'<STR_LIT>' , max_length = <NUM_LIT> ) ) , <EOL> ( '<STR_LIT:image>' , models . ImageField ( help_text = b'<STR_LIT>' , null = True , upload_to = b'<STR_LIT>' , blank = True ) ) , <EOL> ] , <EOL> options = { <EOL> '<STR_LIT>' : ( '<STR_LIT:name>' , ) , <EOL> '<STR_LIT>' : '<STR_LIT>' , <EOL> } , <EOL> bases = ( models . Model , ) , <EOL> ) , <EOL> ] </s>52```53 54 55### Data Statistics56 57 58Data statistics of Github Java Corpus dataset are shown in the below table:59 60| Data Split  |   #Files   |   #Tokens   |61| ----------- | :--------: | :---------: |62|    Train    |   12,934   |    15.7M    |63|     Dev     |    7,176   |     3.8M    |64|    Test     |    8,268   |     5.3M    |65 66 67 68 69