jblitzar/github-python
GitHub-Python — Licensed & Elaborated Variants This repository ships two complementary Python-code corpora extracted from public GitHub: Licensed Subset – strictly permissive-licensed files suitable for commercial redistribution / model training (main corpus used in our experiments). Elaborated Collection – a broader crawl that additionally contains files under copyleft or unclear licenses (GPL/AGPL/LGPL, etc.). Useful for analysis or pre-training where license mixing is… See the full description on the dataset page: https://huggingface.co/datasets/jblitzar/github-python.
0199
1from datasets import Dataset2import pandas as pd3 4a_path = "A_python_files_elaborated_metadata.csv"5b_path = "B_github_python_metadata.csv"6c_path = "C_mega_licensed_corpus_redacted.txt"7 8df_a = pd.read_csv(a_path)9df_a = df_a.rename(columns={10 "owner": "repo_owner",11 "url": "file_url"12})13ds_a = Dataset.from_pandas(df_a)14 15ds_a.push_to_hub("jblitzar/github-python-meta-elaborated", split="train")16 17 18df_b = pd.read_csv("B_github_python_metadata.csv")19df_b["repo_url"] = df_b.get("repo_url", "unknown")20print("Columns in df_b:", df_b.columns)21from datasets import Dataset22ds_b = Dataset.from_pandas(df_b)23ds_b.push_to_hub("jblitzar/github-python-metadata", split="train")24 25 26with open(c_path, encoding="utf-8", errors="ignore") as f:27 lines = f.readlines()28ds_c = Dataset.from_dict({"text": lines})29ds_c.push_to_hub("jblitzar/github-python-corpus", split="train")30