huawei-noah/python_text2code

1---2license: apache-2.03size_categories:4- 10M<n<100M5task_categories:6- text-generation7dataset_info:8  features:9  - name: code10    dtype: string11  - name: docstring12    dtype: string13  - name: _id14    dtype: string15  splits:16  - name: train17    num_bytes: 1875919850218    num_examples: 2352658619  download_size: 854937823820  dataset_size: 1875919850221configs:22- config_name: default23  data_files:24  - split: train25    path: data/train-*26---27 28# Dataset Card for Python-Text2Code29 30This dataset supports the EACL paper [Text-to-Code Generation with Modality-relative Pre-training](https://aclanthology.org/2024.eacl-long.72)31 32- **Repository:** https://github.com/huawei-noah/noah-research/tree/master/NLP/text2code_mrpt33- **Point of Contact:** [Fenia Christopoulou](mailto:efstathia.christopoulou@huawei.com), [Gerasimos Lampouras](mailto:gerasimos.lampouras@huawei.com)34 35## Dataset Description36 37The data were crawled from existing, public repositories from GitHub before May 2021 and were meant to be used for 38additional model training for the task of Code Synthesis (i.e. Text-to-Code generation) in Python.39 40### Details41Files that met the following criteria were kept: 42(a) the file size is under 1MB; 43(b) the code is Python3 compatible, using Abstract Syntactic Tree (AST) parsing; 44(c) there are fewer than 100 characters per line on average; 45(d) and there are fewer than 1,000 characters in any single line. 46 47We applied AST parsing (via [Tree-sitter](https://tree-sitter.github.io/tree-sitter/)) on the remaining Python files to extract valid functions and 48their corresponding docstrings. 49Docstrings were used a "problem descriptions" and were separated from the code. Functions without a docstring were discarded.50We replaced new lines, indentation and dedentation with `<NEW_LINE>`, `<INDENT>` and `<DEDENT>`, respectively, to normalise spaces, which effectively reduced the length51of the sequences.52Finally, only instances with a maximum length of 1024 tokens (docstring+code) were kept. 53 54The final dataset contains 23,526,586 text-to-code pairs in Python.55 56Check the paper for additional details!57 58 59## Data Fields60 61Each instance contains 3 fields:62- `id`: Unique ID of each pair63- `code`: The python code64- `docstring`: The docstring/problem description associated with this code65 66 67## Data Splits68 69There is a single data split in the dataset. We randomly sampled 0.1% of the dataset to serve as validation set.70 71 72## Citation73 74**BibTeX:**75 76```html77@inproceedings{christopoulou-etal-2024-text,78    title = "Text-to-Code Generation with Modality-relative Pre-training",79    author = "Christopoulou, Fenia  and80      Zhang, Guchun  and81      Lampouras, Gerasimos",82    editor = "Graham, Yvette  and83      Purver, Matthew",84    booktitle = "Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)",85    month = mar,86    year = "2024",87    address = "St. Julian{'}s, Malta",88    publisher = "Association for Computational Linguistics",89    url = "https://aclanthology.org/2024.eacl-long.72",90    pages = "1194--1208"91}92 93