jblitzar/github-python

1---2annotations_creators:3  - author4license:5  - gpl-3.06multilinguality:7  - monolingual8pretty_name: GitHub-Python9dataset_name: github-python10dataset_type: code11tags:12  - code13  - python14size_categories:15  - 100K<n⩽1M16task_categories:17  - text-generation18---19 20# GitHub-Python — Licensed & Elaborated Variants21 22This repository ships **two complementary Python-code corpora** extracted from23public GitHub:24 25- **Licensed Subset** – strictly _permissive-licensed_ files suitable for26  commercial redistribution / model training (main corpus used in our27  experiments).28- **Elaborated Collection** – a broader crawl that additionally contains files29  under _copyleft_ or unclear licenses (GPL/AGPL/LGPL, etc.). Useful for30  analysis or pre-training where license mixing is acceptable.31 32Both variants target **code-completion / generation** research.33 34## Dataset at a glance35 36|                     | **Licensed Subset** | **Elaborated Collection** |37| ------------------- | ------------------- | ------------------------- |38| Files (.py)         | 53,017              | 186,066                   |39| Unique repositories | 16,447              | 59,852                    |40| Repository owners   | 12,515              | 43,517                    |41| Compressed size     | 732 MB              | 2.4 GB \*                 |42| Vocabulary (tokens) | 443,431             | 443,431 †                 |43| License coverage    | Permissive only     | Mixed (perm. + copyleft)  |44| Secrets redacted    | ✅                  | ⚠️ not guaranteed         |45| Time window         | ≥ 2015-01-01        | ≥ 2015-01-01              |46 47\* estimated – elaborated corpus is distributed as raw file list, not a single48text file.  49† same tokenizer file is shared by both variants.50 51Numbers were obtained from the final redacted corpus and companion metadata.52 53---54 55## Dataset structure56 57```58huggingface_dataset/59 ├─ mega_licensed_corpus_redacted.txt      # Licensed Subset – concatenated code60 ├─ python_files.txt                       # Licensed Subset – raw file URLs61 ├─ python_files_elaborated.txt            # Elaborated Collection – raw file URLs62 ├─ python_files_elaborated_metadata.csv   # Elaborated Collection metadata63 └─ custom_tokens_vocab.txt             # `<token>\t<id>` vocabulary file64```65 66## Important Note67 68For technical reasons, seperate splits have been stored as seperate Dataset instances. See https://huggingface.co/datasets/jblitzar/github-python-metadata, https://huggingface.co/datasets/jblitzar/github-python-meta-elaborated, and https://huggingface.co/datasets/jblitzar/github-python-corpus .69 70### File separator71 72Individual files are concatenated with the sentinel line:73 74```75# <FILESEP>76```77 78Anything following the sentinel until the next sentinel (or EOF) is the source79code of one file.80 81---82 83## Dataset variants84 85### 1. Licensed Subset (`mega_licensed_corpus_redacted.txt`)86 87• 53 K permissively-licensed files (MIT/BSD/Apache/ISC/Unlicense).  88• All API keys & credentials removed.  89• Ready for redistribution & commercial use (respect upstream NOTICE files).90 91### 2. Elaborated Collection (`python_files_elaborated.txt`)92 93• 186 K files from a much larger crawl.  94• Contains **GPL / LGPL / AGPL and other copyleft** licenses.  95• Shipped _as URL list_ + metadata CSV; you must download the files yourself96(`datasets.load_dataset` streaming, `wget`, etc.).  97• **No license filtering or secret-redaction performed** – use with caution.98 99When first loading the dataset, decide which variant aligns with your use case100(e.g. proprietary model training → Licensed Subset only).101 102 103---104 105## Collection methodology106 1071. **Repository discovery**108 109   - Queried GitHub REST API for projects with **≥ 10 stars**  110     (earlier iterations used 100+, later expanded for coverage).111   - Only repositories with primary language _Python_ and last commit ≥ 2015.112 1132. **File filtering**114 115   - Retain files whose **size ∈ [1 KB, 100 KB]**.116   - Exclude common build/packaging scripts (`setup.py`, `__init__.py`, etc.).117 1183. **License compliance**119 120   - Allowed: MIT, Apache-2.0, BSD-2/3-Clause, ISC, Unlicense.121   - GPL, LGPL, AGPL and proprietary licenses were **excluded**.122 1234. **Deduplication**124 125   - Unique file SHA hashes; duplicates skipped.126 1275. **Formatting & cleaning**128 129   - Formatted with _autopep8_ to normalise whitespace.130   - Custom script removed trailing whitespace & normalised newlines.131 1326. **Secret redaction**133   - `truffleHog` + custom regex pass removed >150 active credentials.134   - Redacted corpus stored as `mega_licensed_corpus_redacted.txt`.135 136---137 138## Custom tokenisation139 140The accompanying `custom_tokens_vocab.txt` implements a **Python-aware141sub-token scheme**:142 1431. Strip doc-strings & comments.1442. Split on:145   - Camel-Case boundaries (`Camel` → `Camel`, `Case`)146   - Underscores, spaces147   - Indentation & newlines (preserved as `<newline>` token)1483. Rare tokens (frequency < 10) were dropped → 443 k vocabulary.149 150Example:151 152```python153def helloWorld(value):154    return value + 1155```156 157tokenises to:158 159```160def hello world ( value ) <newline> <tab> return value + 1 <newline>161```162 163---164 165## Usage166 167```python168from datasets import load_dataset169 170ds = load_dataset("jblitzar/github-python-corpus", split="train")171 172print(ds[0]["code"][:300])        # raw source code173```174 175If you prefer token level examples (small reasons: memory), map the tokenizer:176 177```python178from tokenizers import Tokenizer179tok = Tokenizer.from_file("custom_tokens_vocab.txt")180 181def encode(ex):182    ex["input_ids"] = tok.encode(ex["code"]).ids183    return ex184 185ds = ds.map(encode, remove_columns=["code"])186```187 188---189 190## Ethical considerations & limitations191 192- **Licenses respected** – only permissive licenses included; retain NOTICE193  files when redistributing derivative works.194- **Secrets removed** – automated & manual audits performed, yet users **must195  not assume zero secrets**; re-audit before public deployments.196- **Code quality** – projects vary in style & correctness. Generated models197  may replicate bugs or vulnerable patterns.198 199---200 201## Citation202 203If you use this dataset, please cite:204 205```206@misc{github-python-2024,207  author       = {JBlitzar},208  title        = {GitHub-Python: A Permissively Licensed Corpus of Python Code},209  year         = {2024},210  howpublished = {\url{https://huggingface.co/datasets/jblitzar/github-python}},211  note         = {Version 1.0}212}213```214 215---216 217## License218 219Dataset card and aggregation scripts: **GPLv3**.  220Each code snippet remains under its **original repository license** (MIT,221Apache-2.0, BSD, ISC, etc.). Users must comply with upstream notices when222redistributing code or derivatives.223 224<!--bump-->