codeparrot/github-code

1---2annotations_creators: []3language_creators:4- crowdsourced5- expert-generated6language:7- code8license:9- other10multilinguality:11- multilingual12pretty_name: github-code13size_categories:14- unknown15source_datasets: []16task_categories:17- text-generation18task_ids:19- language-modeling20---21 22# GitHub Code Dataset23 24## Dataset Description25The GitHub Code dataset consists of 115M code files from GitHub in 32 programming languages with 60 extensions totaling in 1TB of data. The dataset was created from the public GitHub dataset on Google BiqQuery.26 27### How to use it28 29The GitHub Code dataset is a very large dataset so for most use cases it is recommended to make use of the streaming API of `datasets`. You can load and iterate through the dataset with the following two lines of code:30 31```python32from datasets import load_dataset33 34ds = load_dataset("codeparrot/github-code", streaming=True, split="train")35print(next(iter(ds)))36 37#OUTPUT:38{39 'code': "import mod189 from './mod189';\nvar value=mod189+1;\nexport default value;\n",40 'repo_name': 'MirekSz/webpack-es6-ts',41 'path': 'app/mods/mod190.js',42 'language': 'JavaScript',43 'license': 'isc',44 'size': 7345}46```47 48You can see that besides the code, repo name, and path also the programming language, license, and the size of the file are part of the dataset. You can also filter the dataset for any subset of the 30 included languages (see the full list below) in the dataset. Just pass the list of languages as a list. E.g. if your dream is to build a Codex model for Dockerfiles use the following configuration:49 50```python51ds = load_dataset("codeparrot/github-code", streaming=True, split="train", languages=["Dockerfile"])52print(next(iter(ds))["code"])53 54#OUTPUT:55"""\56FROM rockyluke/ubuntu:precise57 58ENV DEBIAN_FRONTEND="noninteractive" \59    TZ="Europe/Amsterdam"60...61"""62```63 64We also have access to the license of the origin repo of a file so we can filter for licenses in the same way we filtered for languages:65 66```python67ds = load_dataset("codeparrot/github-code", streaming=True, split="train", licenses=["mit", "isc"])68 69licenses = []70for element in iter(ds).take(10_000):71    licenses.append(element["license"])72print(Counter(licenses))73 74#OUTPUT:75Counter({'mit': 9896, 'isc': 104})76```77 78Naturally, you can also download the full dataset. Note that this will download ~300GB compressed text data and the uncompressed dataset will take up ~1TB of storage:79```python80ds = load_dataset("codeparrot/github-code", split="train")81```82 83## Data Structure84 85### Data Instances86 87```python88{89 'code': "import mod189 from './mod189';\nvar value=mod189+1;\nexport default value;\n",90 'repo_name': 'MirekSz/webpack-es6-ts',91 'path': 'app/mods/mod190.js',92 'language': 'JavaScript',93 'license': 'isc',94 'size': 7395}96```97 98### Data Fields99 100|Field|Type|Description|101|---|---|---|102|code|string|content of source file|103|repo_name|string|name of the GitHub repository|104|path|string|path of file in GitHub repository|105|language|string|programming language as inferred by extension|106|license|string|license of GitHub repository|107|size|int|size of source file in bytes|108 109### Data Splits110 111The dataset only contains a train split.112 113## Languages114 115The dataset contains 30 programming languages with over 60 extensions:116 117```python118{119    "Assembly": [".asm"],120    "Batchfile": [".bat", ".cmd"],121    "C": [".c", ".h"],122    "C#": [".cs"],123    "C++": [".cpp", ".hpp", ".c++", ".h++", ".cc", ".hh", ".C", ".H"],124    "CMake": [".cmake"],125    "CSS": [".css"],126    "Dockerfile": [".dockerfile", "Dockerfile"],127    "FORTRAN": ['.f90', '.f', '.f03', '.f08', '.f77', '.f95', '.for', '.fpp'],128    "GO": [".go"],129    "Haskell": [".hs"],130    "HTML":[".html"],131    "Java": [".java"],132    "JavaScript": [".js"],133    "Julia": [".jl"],134    "Lua": [".lua"],135    "Makefile": ["Makefile"],136    "Markdown": [".md", ".markdown"],137    "PHP": [".php", ".php3", ".php4", ".php5", ".phps", ".phpt"],138    "Perl": [".pl", ".pm", ".pod", ".perl"],139    "PowerShell": ['.ps1', '.psd1', '.psm1'],140    "Python": [".py"],141    "Ruby": [".rb"],142    "Rust": [".rs"],143    "SQL": [".sql"],144    "Scala": [".scala"],145    "Shell": [".sh", ".bash", ".command", ".zsh"],146    "TypeScript": [".ts", ".tsx"],147    "TeX": [".tex"],148    "Visual Basic": [".vb"]149}150```151 152## Licenses153Each example is also annotated with the license of the associated repository. There are in total 15 licenses:154```python155[156  'mit',157  'apache-2.0',158  'gpl-3.0',159  'gpl-2.0',160  'bsd-3-clause',161  'agpl-3.0',162  'lgpl-3.0',163  'lgpl-2.1',164  'bsd-2-clause',165  'cc0-1.0',166  'epl-1.0',167  'mpl-2.0',168  'unlicense',169  'isc',170  'artistic-2.0'171 ]172```173 174## Dataset Statistics175 176The dataset contains 115M files and the sum of all the source code file sizes is 873 GB (note that the size of the dataset is larger due to the extra fields). A breakdown per language is given in the plot and table below:177 178![dataset-statistics](https://huggingface.co/datasets/codeparrot/github-code/resolve/main/github-code-stats-alpha.png)179 180|    | Language     |File Count| Size (GB)|181|---:|:-------------|---------:|-------:|182|  0 | Java         | 19548190 | 107.70 |183|  1 | C            | 14143113 | 183.83 |184|  2 | JavaScript   | 11839883 |  87.82 |185|  3 | HTML         | 11178557 | 118.12 |186|  4 | PHP          | 11177610 |  61.41 |187|  5 | Markdown     |  8464626 |  23.09 |188|  6 | C++          |  7380520 |  87.73 |189|  7 | Python       |  7226626 |  52.03 |190|  8 | C#           |  6811652 |  36.83 |191|  9 | Ruby         |  4473331 |  10.95 |192| 10 | GO           |  2265436 |  19.28 |193| 11 | TypeScript   |  1940406 |  24.59 |194| 12 | CSS          |  1734406 |  22.67 |195| 13 | Shell        |  1385648 |   3.01 |196| 14 | Scala        |   835755 |   3.87 |197| 15 | Makefile     |   679430 |   2.92 |198| 16 | SQL          |   656671 |   5.67 |199| 17 | Lua          |   578554 |   2.81 |200| 18 | Perl         |   497949 |   4.70 |201| 19 | Dockerfile   |   366505 |   0.71 |202| 20 | Haskell      |   340623 |   1.85 |203| 21 | Rust         |   322431 |   2.68 |204| 22 | TeX          |   251015 |   2.15 |205| 23 | Batchfile    |   236945 |   0.70 |206| 24 | CMake        |   175282 |   0.54 |207| 25 | Visual Basic |   155652 |   1.91 |208| 26 | FORTRAN      |   142038 |   1.62 |209| 27 | PowerShell   |   136846 |   0.69 |210| 28 | Assembly     |    82905 |   0.78 |211| 29 | Julia        |    58317 |   0.29 |212 213 214## Dataset Creation215 216The dataset was created in two steps:2171. Files of with the extensions given in the list above were retrieved from the GitHub dataset on BigQuery (full query [here](https://huggingface.co/datasets/codeparrot/github-code/blob/main/query.sql)). The query was executed on _Mar 16, 2022, 6:23:39 PM UTC+1_.2182. Files with lines longer than 1000 characters and duplicates (exact duplicates ignoring whitespaces) were dropped (full preprocessing script [here](https://huggingface.co/datasets/codeparrot/github-code/blob/main/github_preprocessing.py)).219 220## Considerations for Using the Data221 222The dataset consists of source code from a wide range of repositories. As such they can potentially include harmful or biased code as well as sensitive information like passwords or usernames.223 224## Releases225 226You can load any older version of the dataset with the `revision` argument:227 228```Python229ds = load_dataset("codeparrot/github-code", revision="v1.0")230```231 232### v1.0233- Initial release of dataset234- The query was executed on _Feb 14, 2022, 12:03:16 PM UTC+1_235 236### v1.1237- Fix missing Scala/TypeScript238- Fix deduplication issue with inconsistent Python `hash`239- The query was executed on _Mar 16, 2022, 6:23:39 PM UTC+1_240