AlgorithmicResearchGroup/arxiv_python_research_code

1---2dataset_info:3  features:4  - name: repo5    dtype: string6  - name: file7    dtype: string8  - name: code9    dtype: string10  - name: file_length11    dtype: int6412  - name: avg_line_length13    dtype: float6414  - name: max_line_length15    dtype: int6416  - name: extension_type17    dtype: string18  splits:19  - name: train20    num_bytes: 1298419977821    num_examples: 141592422  download_size: 407385361623  dataset_size: 1298419977824license: bigcode-openrail-m25task_categories:26- text-generation27language:28- en29pretty_name: arxiv_python_research_code30size_categories:31- 1B<n<10B32---33# Dataset Card for "ArtifactAI/arxiv_python_research_code"34 35## Dataset Description36 37https://huggingface.co/datasets/AlgorithmicResearchGroup/arxiv_python_research_code38 39 40### Dataset Summary41 42AlgorithmicResearchGroup/arxiv_python_research_code contains over 4.13GB of  source code files referenced strictly in ArXiv papers. The dataset serves as a curated dataset for Code LLMs.43 44### How to use it45```python46from datasets import load_dataset47 48# full dataset (4.13GB of data)49ds = load_dataset("AlgorithmicResearchGroup/arxiv_python_research_code", split="train")50 51# dataset streaming (will only download the data as needed)52ds = load_dataset("AlgorithmicResearchGroup/arxiv_python_research_code", streaming=True, split="train")53for sample in iter(ds): print(sample["code"])54```55 56## Dataset Structure57### Data Instances58Each data instance corresponds to one file. The content of the file is in the `code` feature, and other features (`repo`, `file`, etc.) provide some metadata.59### Data Fields60- `repo` (string): code repository name.61- `file` (string): file path in the repository.62- `code` (string): code within the file.63- `file_length`: (integer): number of characters in the file.64- `avg_line_length`: (float): the average line-length of the file.65- `max_line_length`: (integer): the maximum line-length of the file.66- `extension_type`: (string): file extension.67 68### Data Splits69 70The dataset has no splits and all data is loaded as train split by default.71 72## Dataset Creation73 74### Source Data75#### Initial Data Collection and Normalization7634,099 active GitHub repository names were extracted from [ArXiv](https://arxiv.org/) papers from its inception through July 21st, 2023 totaling 773G of compressed github repositories.77 78These repositories were then filtered, and the code from each '.py' file extension was extracted into 1.4 million files.79 80#### Who are the source language producers?81 82The source (code) language producers are users of GitHub that created unique repository83 84### Personal and Sensitive Information85The released dataset may contain sensitive information such as emails, IP addresses, and API/ssh keys that have previously been published to public repositories on GitHub. 86 87## Additional Information88 89### Dataset Curators90Matthew Kenney, AlgorithmicResearchGroup, matt@algorithmicresearchgroup.com91 92### Citation Information93```94@misc{arxiv_python_research_code,95    title={arxiv_python_research_code},96    author={Matthew Kenney},97    year={2023}98}99```