TheFinAI/github-java-corpus

github-java-corpus

Summary

This dataset contains Java source-code text samples prepared for pretraining.

Repository

TheFinAI/github-java-corpus

Required Columns

• —Source: dataset name
• —Date: year
• —Text: the pure text of each sample
• —Token_count: the token count computed with tiktoken

Schema

• —Source (string)
• —Date (int32)
• —Text (string)
• —Token_count (int32)

Construction

The dataset was built from streamed archive processing into file-level text samples.

Notes

• —Intended for pretraining
• —One row corresponds to one source file
• —Date is the assigned year label for this source

Source Dataset

This dataset is derived from the Project CodeNet dataset:

https://datashare.ed.ac.uk/handle/10283/2334

Project CodeNet is a large-scale dataset of source code collected from online programming platforms, including AIZU Online Judge and AtCoder.

Processing

We convert the original dataset into a file-level text corpus suitable for pretraining:

• —Extract Java source files
• —Remove generated files
• —Convert into JSONL format
• —Compute token counts using tiktoken
• —Upload as a Hugging Face dataset

Citation

If you use this dataset, please also cite the original Project CodeNet dataset:

@inproceedings{puri2021project, title={Project CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks}, author={Puri, Ruchir et al.}, booktitle={NeurIPS Datasets and Benchmarks}, year={2021} }