code-search-net/code_search_net

1---2annotations_creators:3- no-annotation4language_creators:5- machine-generated6language:7- code8license:9- other10multilinguality:11- multilingual12size_categories:13- 100K<n<1M14- 10K<n<100K15- 1M<n<10M16source_datasets:17- original18task_categories:19- text-generation20- fill-mask21task_ids:22- language-modeling23- masked-language-modeling24paperswithcode_id: codesearchnet25pretty_name: CodeSearchNet26dataset_info:27- config_name: all28  features:29  - name: repository_name30    dtype: string31  - name: func_path_in_repository32    dtype: string33  - name: func_name34    dtype: string35  - name: whole_func_string36    dtype: string37  - name: language38    dtype: string39  - name: func_code_string40    dtype: string41  - name: func_code_tokens42    sequence: string43  - name: func_documentation_string44    dtype: string45  - name: func_documentation_tokens46    sequence: string47  - name: split_name48    dtype: string49  - name: func_code_url50    dtype: string51  splits:52  - name: train53    num_bytes: 585060408354    num_examples: 188085355  - name: test56    num_bytes: 30862633357    num_examples: 10052958  - name: validation59    num_bytes: 27456438260    num_examples: 8915461  download_size: 511737051162  dataset_size: 643379479863- config_name: java64  features:65  - name: repository_name66    dtype: string67  - name: func_path_in_repository68    dtype: string69  - name: func_name70    dtype: string71  - name: whole_func_string72    dtype: string73  - name: language74    dtype: string75  - name: func_code_string76    dtype: string77  - name: func_code_tokens78    sequence: string79  - name: func_documentation_string80    dtype: string81  - name: func_documentation_tokens82    sequence: string83  - name: split_name84    dtype: string85  - name: func_code_url86    dtype: string87  splits:88  - name: train89    num_bytes: 142927253590    num_examples: 45445191  - name: test92    num_bytes: 8237724693    num_examples: 2690994  - name: validation95    num_bytes: 4235831596    num_examples: 1532897  download_size: 106056915398  dataset_size: 155400809699- config_name: go100  features:101  - name: repository_name102    dtype: string103  - name: func_path_in_repository104    dtype: string105  - name: func_name106    dtype: string107  - name: whole_func_string108    dtype: string109  - name: language110    dtype: string111  - name: func_code_string112    dtype: string113  - name: func_code_tokens114    sequence: string115  - name: func_documentation_string116    dtype: string117  - name: func_documentation_tokens118    sequence: string119  - name: split_name120    dtype: string121  - name: func_code_url122    dtype: string123  splits:124  - name: train125    num_bytes: 738153234126    num_examples: 317832127  - name: test128    num_bytes: 32286998129    num_examples: 14291130  - name: validation131    num_bytes: 26888527132    num_examples: 14242133  download_size: 487525935134  dataset_size: 797328759135- config_name: python136  features:137  - name: repository_name138    dtype: string139  - name: func_path_in_repository140    dtype: string141  - name: func_name142    dtype: string143  - name: whole_func_string144    dtype: string145  - name: language146    dtype: string147  - name: func_code_string148    dtype: string149  - name: func_code_tokens150    sequence: string151  - name: func_documentation_string152    dtype: string153  - name: func_documentation_tokens154    sequence: string155  - name: split_name156    dtype: string157  - name: func_code_url158    dtype: string159  splits:160  - name: train161    num_bytes: 1559645310162    num_examples: 412178163  - name: test164    num_bytes: 84342064165    num_examples: 22176166  - name: validation167    num_bytes: 92154786168    num_examples: 23107169  download_size: 940909997170  dataset_size: 1736142160171- config_name: javascript172  features:173  - name: repository_name174    dtype: string175  - name: func_path_in_repository176    dtype: string177  - name: func_name178    dtype: string179  - name: whole_func_string180    dtype: string181  - name: language182    dtype: string183  - name: func_code_string184    dtype: string185  - name: func_code_tokens186    sequence: string187  - name: func_documentation_string188    dtype: string189  - name: func_documentation_tokens190    sequence: string191  - name: split_name192    dtype: string193  - name: func_code_url194    dtype: string195  splits:196  - name: train197    num_bytes: 480286523198    num_examples: 123889199  - name: test200    num_bytes: 24056972201    num_examples: 6483202  - name: validation203    num_bytes: 30168242204    num_examples: 8253205  download_size: 1664713350206  dataset_size: 534511737207- config_name: ruby208  features:209  - name: repository_name210    dtype: string211  - name: func_path_in_repository212    dtype: string213  - name: func_name214    dtype: string215  - name: whole_func_string216    dtype: string217  - name: language218    dtype: string219  - name: func_code_string220    dtype: string221  - name: func_code_tokens222    sequence: string223  - name: func_documentation_string224    dtype: string225  - name: func_documentation_tokens226    sequence: string227  - name: split_name228    dtype: string229  - name: func_code_url230    dtype: string231  splits:232  - name: train233    num_bytes: 110681715234    num_examples: 48791235  - name: test236    num_bytes: 5359280237    num_examples: 2279238  - name: validation239    num_bytes: 4830744240    num_examples: 2209241  download_size: 111758028242  dataset_size: 120871739243- config_name: php244  features:245  - name: repository_name246    dtype: string247  - name: func_path_in_repository248    dtype: string249  - name: func_name250    dtype: string251  - name: whole_func_string252    dtype: string253  - name: language254    dtype: string255  - name: func_code_string256    dtype: string257  - name: func_code_tokens258    sequence: string259  - name: func_documentation_string260    dtype: string261  - name: func_documentation_tokens262    sequence: string263  - name: split_name264    dtype: string265  - name: func_code_url266    dtype: string267  splits:268  - name: train269    num_bytes: 1532564870270    num_examples: 523712271  - name: test272    num_bytes: 80203877273    num_examples: 28391274  - name: validation275    num_bytes: 78163924276    num_examples: 26015277  download_size: 851894048278  dataset_size: 1690932671279config_names:280- all281- go282- java283- javascript284- php285- python286- ruby287---288 289# Dataset Card for CodeSearchNet corpus290 291## Table of Contents292- [Dataset Description](#dataset-description)293  - [Dataset Summary](#dataset-summary)294  - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)295  - [Languages](#languages)296- [Dataset Structure](#dataset-structure)297  - [Data Instances](#data-instances)298  - [Data Fields](#data-fields)299  - [Data Splits](#data-splits)300- [Dataset Creation](#dataset-creation)301  - [Curation Rationale](#curation-rationale)302  - [Source Data](#source-data)303  - [Annotations](#annotations)304  - [Personal and Sensitive Information](#personal-and-sensitive-information)305- [Considerations for Using the Data](#considerations-for-using-the-data)306  - [Social Impact of Dataset](#social-impact-of-dataset)307  - [Discussion of Biases](#discussion-of-biases)308  - [Other Known Limitations](#other-known-limitations)309- [Additional Information](#additional-information)310  - [Dataset Curators](#dataset-curators)311  - [Licensing Information](#licensing-information)312  - [Citation Information](#citation-information)313  - [Contributions](#contributions)314 315## Dataset Description316- **Homepage:** https://wandb.ai/github/CodeSearchNet/benchmark317- **Repository:** https://github.com/github/CodeSearchNet318- **Paper:** https://arxiv.org/abs/1909.09436319- **Data:** https://doi.org/10.5281/zenodo.7908468320- **Leaderboard:** https://wandb.ai/github/CodeSearchNet/benchmark/leaderboard321 322### Dataset Summary323 324CodeSearchNet corpus is a dataset of 2 milllion (comment, code) pairs from opensource libraries hosted on GitHub. It contains code and documentation for several programming languages.325 326CodeSearchNet corpus was gathered to support the [CodeSearchNet challenge](https://wandb.ai/github/CodeSearchNet/benchmark), to explore the problem of code retrieval using natural language.327 328### Supported Tasks and Leaderboards329 330- `language-modeling`: The dataset can be used to train a model for modelling programming languages, which consists in building language models for programming languages.331 332### Languages333 334- Go **programming** language335- Java **programming** language336- Javascript **programming** language337- PHP **programming** language338- Python **programming** language339- Ruby **programming** language340 341## Dataset Structure342 343### Data Instances344 345A data point consists of a function code along with its documentation. Each data point also contains meta data on the function, such as the repository it was extracted from.346```347{348  'id': '0',349  'repository_name': 'organisation/repository',350  'func_path_in_repository': 'src/path/to/file.py',351  'func_name': 'func',352  'whole_func_string': 'def func(args):\n"""Docstring"""\n [...]',353  'language': 'python', 354  'func_code_string': '[...]',355  'func_code_tokens': ['def', 'func', '(', 'args', ')', ...],356  'func_documentation_string': 'Docstring',357  'func_documentation_string_tokens': ['Docstring'],358  'split_name': 'train',359  'func_code_url': 'https://github.com/<org>/<repo>/blob/<hash>/src/path/to/file.py#L111-L150'360}361```362### Data Fields363 364- `id`: Arbitrary number365- `repository_name`: name of the GitHub repository366- `func_path_in_repository`: tl;dr: path to the file which holds the function in the repository367- `func_name`: name of the function in the file368- `whole_func_string`: Code + documentation of the function369- `language`: Programming language in whoch the function is written370- `func_code_string`: Function code371- `func_code_tokens`: Tokens yielded by Treesitter372- `func_documentation_string`: Function documentation373- `func_documentation_string_tokens`: Tokens yielded by Treesitter374- `split_name`: Name of the split to which the example belongs (one of train, test or valid)375- `func_code_url`: URL to the function code on Github376 377### Data Splits378 379Three splits are available:380- train381- test382- valid383 384## Dataset Creation385 386### Curation Rationale387 388[More Information Needed]389 390### Source Data391 392#### Initial Data Collection and Normalization393 394All information can be retrieved in the [original technical review](https://arxiv.org/pdf/1909.09436.pdf)395 396**Corpus collection**:397 398Corpus has been collected from publicly available open-source non-fork GitHub repositories, using libraries.io to identify all projects which are used by at least one other project, and sort them by “popularity” as indicated by the number of stars and forks. 399 400Then, any projects that do not have a license or whose license does not explicitly permit the re-distribution of parts of the project were removed. Treesitter - GitHub's universal parser - has been used to then tokenize all Go, Java, JavaScript, Python, PHP and Ruby functions (or methods) using and, where available, their respective documentation text using a heuristic regular expression.401 402**Corpus filtering**:403 404Functions without documentation are removed from the corpus. This yields a set of pairs ($c_i$, $d_i$) where ci is some function documented by di. Pairs ($c_i$, $d_i$) are passed through the folllowing preprocessing tasks:405 406- Documentation $d_i$ is truncated to the first full paragraph to remove in-depth discussion of function arguments and return values407- Pairs in which $d_i$ is shorter than three tokens are removed408- Functions $c_i$ whose implementation is shorter than three lines are removed409- Functions whose name contains the substring “test” are removed410- Constructors and standard extenion methods (eg `__str__` in Python or `toString` in Java) are removed411- Duplicates and near duplicates functions are removed, in order to keep only one version of the function412 413#### Who are the source language producers?414 415OpenSource contributors produced the code and documentations.416 417The dataset was gatherered and preprocessed automatically.418 419### Annotations420 421#### Annotation process422 423[More Information Needed]424 425#### Who are the annotators?426 427[More Information Needed]428 429### Personal and Sensitive Information430 431[More Information Needed]432 433## Considerations for Using the Data434 435### Social Impact of Dataset436 437[More Information Needed]438 439### Discussion of Biases440 441[More Information Needed]442 443### Other Known Limitations444 445[More Information Needed]446 447## Additional Information448 449### Dataset Curators450 451[More Information Needed]452 453### Licensing Information454 455Each example in the dataset has is extracted from a GitHub repository, and each repository has its own license. Example-wise license information is not (yet) included in this dataset: you will need to find out yourself which license the code is using.456 457### Citation Information458 459@article{husain2019codesearchnet,460  title={{CodeSearchNet} challenge: Evaluating the state of semantic code search},461  author={Husain, Hamel and Wu, Ho-Hsiang and Gazit, Tiferet and Allamanis, Miltiadis and Brockschmidt, Marc},462  journal={arXiv preprint arXiv:1909.09436},463  year={2019}464}465 466### Contributions467 468Thanks to [@SBrandeis](https://github.com/SBrandeis) for adding this dataset.469