codeparrot/github-code

1---2annotations_creators: []3language_creators:4- crowdsourced5- expert-generated6languages: []7licenses:8- other-multiple9multilinguality:10- multilingual11pretty_name: github-code12size_categories:13- unknown14source_datasets: []15task_categories:16- sequence-modeling17task_ids:18- language-modeling19---20 21# GitHub Code Dataset22 23## Dataset Description24The GitHub Code dataest consists of 115M code files from GitHub in 32 programming languages with 60 extensions totalling in 1TB of text data. The dataset was created from the public GitHub dataset on Google BiqQuery.25 26### How to use it27 28The GitHub Code dataset is a very large dataset so for most use cases it is recommended to make use of the streaming API of `datasets`. You can load and iterate through the dataset with the following two lines of code:29 30```python31from datasets import load_dataset32 33ds = load_dataset("lvwerra/github-code", streaming=True, split="train")34print(next(iter(ds)))35 36#OUTPUT:37{38 'code': "import mod189 from './mod189';\nvar value=mod189+1;\nexport default value;\n",39 'repo_name': 'MirekSz/webpack-es6-ts',40 'path': 'app/mods/mod190.js',41 'language': 'JavaScript',42 'license': 'isc',43 'size': 7344}45```46 47You can see that besides the code, repo name, and path also the programming language, license, and the size of the file are part of the dataset. You can also filter the dataset for any subset of the 30 included languages (see the full list below) in the dataset. Just pass the list of languages as a list. E.g. if your dream is to build a Codex model for Dockerfiles use the following configuration:48 49```python50ds = load_dataset("lvwerra/github-code", streaming=True, split="train", languages=["Dockerfile"])51print(next(iter(ds))["code"])52 53#OUTPUT:54"""\55FROM rockyluke/ubuntu:precise56 57ENV DEBIAN_FRONTEND="noninteractive" \58    TZ="Europe/Amsterdam"59...60"""61```62 63We also have access to the license of the origin repo of a file so we can filter for licenses in the same way we filtered for languages:64 65```python66ds = load_dataset("lvwerra/github-code", streaming=True, split="train", licenses=["mit", "isc"])67 68licenses = []69for element in iter(ds).take(10_000):70    licenses.append(element["license"])71print(Counter(licenses))72 73#OUTPUT:74Counter({'mit': 9896, 'isc': 104})75```76 77Naturally, you can also download the full dataset. Note that this will download ~300GB compressed text data and the uncompressed dataset will take up ~1TB of storage:78```python79ds = load_dataset("lvwerra/github-code", split="train")80```81 82## Data Structure83 84### Data Instances85 86```python87{88 'code': "import mod189 from './mod189';\nvar value=mod189+1;\nexport default value;\n",89 'repo_name': 'MirekSz/webpack-es6-ts',90 'path': 'app/mods/mod190.js',91 'language': 'JavaScript',92 'license': 'isc',93 'size': 7394}95```96 97### Data Fields98 99|Field|Type|Description|100|---|---|---|101|code|string|content of source file|102|repo_name|string|name of the GitHub repository|103|path|string|path of file in GitHub repository|104|language|string|programming language as inferred by extension|105|license|string|license of GitHub repository|106|size|int|size of source file in bytes|107 108### Data Splits109 110The dataset only contains a train split.111 112## Languages113 114The dataset contains 30 programming languages with over 60 extensions:115 116```python117{118    "Assembly": [".asm"],119    "Batchfile": [".bat", ".cmd"],120    "C": [".c", ".h"],121    "C#": [".cs"],122    "C++": [".cpp", ".hpp", ".c++", ".h++", ".cc", ".hh", ".C", ".H"],123    "CMake": [".cmake"],124    "CSS": [".css"],125    "Dockerfile": [".dockerfile", "Dockerfile"],126    "FORTRAN": ['.f90', '.f', '.f03', '.f08', '.f77', '.f95', '.for', '.fpp'],127    "GO": [".go"],128    "Haskell": [".hs"],129    "HTML":[".html"],130    "Java": [".java"],131    "JavaScript": [".js"],132    "Julia": [".jl"],133    "Lua": [".lua"],134    "Makefile": ["Makefile"],135    "Markdown": [".md", ".markdown"],136    "PHP": [".php", ".php3", ".php4", ".php5", ".phps", ".phpt"],137    "Perl": [".pl", ".pm", ".pod", ".perl"],138    "PowerShell": ['.ps1', '.psd1', '.psm1'],139    "Python": [".py"],140    "Ruby": [".rb"],141    "Rust": [".rs"],142    "SQL": [".sql"],143    "Scala": [".scala"],144    "Shell": [".sh", ".bash", ".command", ".zsh"],145    "TypeScript": [".ts", ".tsx"],146    "TeX": [".tex"],147    "Visual Basic": [".vb"]148}149```150 151## Licenses152Each example is also annotated with the license of the associated repository. There are in total 15 licenses:153```python154[155  'mit',156  'apache-2.0',157  'gpl-3.0',158  'gpl-2.0',159  'bsd-3-clause',160  'agpl-3.0',161  'lgpl-3.0',162  'lgpl-2.1',163  'bsd-2-clause',164  'cc0-1.0',165  'epl-1.0',166  'mpl-2.0',167  'unlicense',168  'isc',169  'artistic-2.0'170 ]171```172 173## Dataset Creation174 175The dataset was created in two steps:1761. Files of with the extensions given in the list above were retrieved from the GitHub dataset on BigQuery (full query [here](https://huggingface.co/datasets/lvwerra/github-code/blob/main/query.sql)). The query was executed on _Feb 14, 2022, 12:03:16 PM UTC+1_.1772. Files with lines longer than 1000 characters and duplicates (exact duplicates ignoring whitespaces) were dropped (full preprocessing script [here](https://huggingface.co/datasets/lvwerra/github-code/blob/main/github_preprocessing.py)).178 179## Considerations for Using the Data180 181The dataset consists of source code from a wide range of repositories. As such they can potentially include harmful or biased code as well as sensitive information like passwords or usernames.