TheFinAI/github-java-corpus

1---2dataset_info:3  features:4  - name: Source5    dtype: string6  - name: Date7    dtype: int328  - name: Text9    dtype: string10  - name: Token_count11    dtype: int3212  splits:13  - name: train14    num_bytes: 1188816260115    num_examples: 211889016  download_size: 309067945017  dataset_size: 1188816260118configs:19- config_name: default20  data_files:21  - split: train22    path: data/train-*23pretty_name: github-java-corpus24language:25- code26license: other27task_categories:28- text-generation29tags:30- text31- code32- java33---34 35# github-java-corpus36 37## Summary38This dataset contains Java source-code text samples prepared for pretraining.39 40## Repository41`TheFinAI/github-java-corpus`42 43## Required Columns44- **Source**: dataset name45- **Date**: year46- **Text**: the pure text of each sample47- **Token_count**: the token count computed with tiktoken48 49## Schema50- `Source` (`string`)51- `Date` (`int32`)52- `Text` (`string`)53- `Token_count` (`int32`)54 55## Construction56The dataset was built from streamed archive processing into file-level text samples.57 58## Notes59- Intended for pretraining60- One row corresponds to one source file61- `Date` is the assigned year label for this source62 63## Source Dataset64 65This dataset is derived from the Project CodeNet dataset:66 67https://datashare.ed.ac.uk/handle/10283/233468 69Project CodeNet is a large-scale dataset of source code collected from online programming platforms, including AIZU Online Judge and AtCoder.70 71## Processing72 73We convert the original dataset into a file-level text corpus suitable for pretraining:74 75- Extract Java source files76- Remove generated files77- Convert into JSONL format78- Compute token counts using tiktoken79- Upload as a Hugging Face dataset80 81## Citation82 83If you use this dataset, please also cite the original Project CodeNet dataset:84 85@inproceedings{puri2021project,86  title={Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks},87  author={Puri, Ruchir et al.},88  booktitle={NeurIPS Datasets and Benchmarks},89  year={2021}90}