codeparrot/github-code-clean
The GitHub Code clean dataset in a more filtered version of codeparrot/github-code dataset, it consists of 115M code files from GitHub in 32 programming languages with 60 extensions totaling in almost 1TB of text data.
14325k
This is a cleaner version of Github-code dataset, we add the following filters:
- Average line length < 100
- Alpha numeric characters fraction > 0.25
- Remove auto-generated files (keyword search)
3.39M files are removed making up 2.94% of the dataset.