codeparrot/code-generation-models

1[InCoder](https://huggingface.co/facebook/incoder-6B) is a code generation model that also allows code editing via [infilling](https://arxiv.org/pdf/2204.05999.pdf). It was trained on **216 GB** of preprocessed data from GitHub and Stack Overflow from 28 programming languages. 52 GB is in Python, 107GB in other programming languages and 57GB is content from Stackoverflow that isn't code.2 3The GitHub data was cleaned with the following steps:4- Average line length < 100 tokens5- Maximum line length < 3000 MB6- Alphanumeric characters fraction > 0.4 7- Remove auto-generated files (keyword search)8 9The second component of the data consists of questions, answers, and comments from Stack Overflow. It includes:10- all questions that have at least one answer11- up to ten answers with a non-negative score (sorted by score) per question12- up to five comments per question/answer13 14Exact match deduplication was performed on code files. For more details please refer to this [paper](https://arxiv.org/pdf/2204.05999.pdf).