shibing624/source_code

1---2annotations_creators:3- no-annotation4language_creators:5- crowdsourced6language:7- en8license:9- cc-by-4.010- gfdl11multilinguality:12- monolingual13size_categories:14- 100M<n<200M15source_datasets:16- https://github.com/shibing624/code-autocomplete17- https://github.com/bharathgs/Awesome-pytorch-list18- https://github.com/akullpp/awesome-java19- https://github.com/fffaraz/awesome-cpp20task_categories:21- text-generation22task_ids:23- language-modeling24---25# Dataset Card for "SourceCode"26## Table of Contents27- [Dataset Description](#dataset-description)28  - [Dataset Summary](#dataset-summary)29  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)30  - [Languages](#languages)31- [Dataset Structure](#dataset-structure)32  - [Data Instances](#data-instances)33  - [Data Fields](#data-fields)34  - [Data Splits](#data-splits)35- [Dataset Creation](#dataset-creation)36  - [Curation Rationale](#curation-rationale)37  - [Source Data](#source-data)38  - [Annotations](#annotations)39  - [Personal and Sensitive Information](#personal-and-sensitive-information)40- [Considerations for Using the Data](#considerations-for-using-the-data)41  - [Social Impact of Dataset](#social-impact-of-dataset)42  - [Discussion of Biases](#discussion-of-biases)43  - [Other Known Limitations](#other-known-limitations)44- [Additional Information](#additional-information)45  - [Dataset Curators](#dataset-curators)46  - [Licensing Information](#licensing-information)47  - [Citation Information](#citation-information)48  - [Contributions](#contributions)49 50## Dataset Description51- **Repository:** [code-autocomplete](https://github.com/shibing624/code-autocomplete)52- **Leaderboard:** [leaderboard](https://github.com/shibing624/code-autocomplete) (located on the homepage)53- **Size of downloaded dataset files:** 105 MB54- **Total amount of disk used:** 570 MB55 56### Dataset Summary57 58Source code dataset is a collection of Github awesome repos, it contains Python, Java, C++, and other programming languages.59This dataset can be used in different NLP tasks like language modeling and text generation tasks.60 61data source:62 63- PYTHON_CODE: https://github.com/bharathgs/Awesome-pytorch-list64- JAVA_CODE: https://github.com/akullpp/awesome-java65- CPP_CODE: https://github.com/fffaraz/awesome-cpp66 67 68### Supported Tasks and Leaderboards69- language modeling 70- code generation tasks, **Leaderboard:** [code-autocomplete](https://github.com/shibing624/code-autocomplete)71 72### Languages73 74- programming languages: Python, Java, C++75- natural language: English76 77## Dataset Structure78### Data Instances79An example of 'train' looks as follows.80```81This example was too long and was cropped:82 83{84    "text": """85import json86import argparse87 88 89def _parse_args():90    parser = argparse.ArgumentParser(91        description=__doc__,92        formatter_class=argparse.RawTextHelpFormatter,93    )94    parser.add_argument(95        '--model-file',96        required=True,97        help=(98            'A pt file from '99            'https://github.com/pytorch/fairseq/tree/main/examples/hubert'100        )101    )102    return parser.parse_args()103    """104}105```106### Data Fields107The data fields are the same among all splits.108- `text`: a `string` feature.109### Data Splits110#### python111```shell112$ wc -l python/*113   10000 python/test.txt114 5215412 python/train.txt115   10000 python/valid.txt116 5235412 total117```118#### java119```shell120$ wc -l java/*  121  950083 java/test.txt122 2802880 java/train.txt123  940803 java/valid.txt124 4693766 total125```126#### cpp127```shell128$ wc -l cpp/* 129 1060014 cpp/test.txt130 3119241 cpp/train.txt131 1099124 cpp/valid.txt132 5278379 total133```134## Dataset Creation135### Curation Rationale136As code generation dataset, I upload it to huggingface datasets.137### Source Data138#### Initial Data Collection and Normalization139#### Who are the source language producers?140Citation:141 142APA:143```latex144Xu, M. code-autocomplete: Code AutoComplete with GPT2 model (Version 0.0.4) [Computer software]. https://github.com/shibing624/code-autocomplete145```146 147BibTeX:148```latex149@software{Xu_code-autocomplete_Code_AutoComplete,150author = {Xu, Ming},151title = {code-autocomplete: Code AutoComplete with GPT2 model},152url = {https://github.com/shibing624/code-autocomplete},153version = {0.0.4}154}155```156 157### Annotations158#### Annotation process159#### Who are the annotators?160nobody161### Personal and Sensitive Information162## Considerations for Using the Data163### Social Impact of Dataset164This dataset was developed as a benchmark for evaluating code generation model.165### Discussion of Biases166### Other Known Limitations167## Additional Information168### Dataset Curators169 170Github awesome programing code repos.171 172### Licensing Information173 174GNU Free Documentation License v1.3 or later.175 176For research use only.177 178### Contributions179Thanks to [@shibing624](https://github.com/shibing624) add this dataset.180