CarperAI/diff-codegen-6b-v2

1---2language: 3  - en4  - code5  6license: "mit"7 8tags:9- Diff Model10- pytorch11- causal-lm12- code-generation13- The Pile14---15 16# Diff-Codegen-6B v2 Model Card17 18## Model Description19 20diff-codegen-6b-v2 is a diff model for code generation, released by [CarperAI](http://carper.ai/). A diff model is an autoregressive language model trained on edits to a piece of text, formatted in [Unified Diff Format](https://en.wikipedia.org/wiki/Diff#Unified_format). These diff models can suggest, given a section of text and a description of the desired change, an intelligent change to the text that fits the description, marking the lines added, changed, and deleted in diff format.21 22In comparison to few-shot prompting of normal code generation models, diff models are specialized for suggesting intelligent changes to existing code, particularly longer pieces of code and where a change is required to follow some natural language text description (provided in the form of a commit message).23 24This model is a fine-tune of [codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) by Salesforce, trained on a large dataset of commits scraped from GitHub.25 26diff-codegen-6b-v2 is an experimental research artifact and should be treated as such. We are releasing these results and this model in the hopes that it may be useful to the greater research community, especially those interested in LMs for code.27 28An example Colab notebook with a brief example of prompting the model is [here](https://colab.research.google.com/drive/1ySm6HYvALerDiGmk6g3pDz68V7fAtrQH#scrollTo=thvzNpmahNNx).29 30## Training Data31 32This model is a fine-tune of [codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) by Salesforce. This language model was first pre-trained on The Pile, an 800Gb dataset composed of varied web corpora. The datasheet and paper for the Pile can be found [here](https://arxiv.org/abs/2201.07311) and [here](https://arxiv.org/abs/2101.00027) respectively. The model was then fine-tuned on a large corpus of code data in multiple languages, before finally being fine-tuned on a Python code dataset. The Codegen paper with full details of these datasets can be found [here](https://arxiv.org/abs/2203.13474).33 34Our dataset for this fine-tune consists of commits from GitHub, obtained using the [Google BigQuery Public Dataset](https://cloud.google.com/blog/topics/public-datasets/github-on-bigquery-analyze-all-the-open-source-code), a public up to date snapshot of a huge number of open-source GitHub repositories. We took this dataset and filtered using [BigQuery](https://console.cloud.google.com/marketplace/details/github/github-repos) on the number of stars in the repository to exclude repos with less than 100 stars, and further restricted the query to only repositories with open-source non-copyleft licenses (e.g. MIT, Apache, etc) and commits with more than 10 characters in the commit message. We also restricted ourselves to a list of 22 popular programming, scripting, and markup languages, including Python, HTML, Bash scripts, SQL, C++, etc. This resulted in a dataset of 19 million commits after filtering.35 36Our diff model was trained on a dataset of commits from BigQuery, a large-scale dataset of many programming languages from GitHub repositories. We filtered the dataset by the number of stars in the repository (>100 stars), license (only open-source non-copyleft licensed code included), and length of file (files greater than 2048 tokens in length were excluded).37 38The model was trained using the Huggingface Codegen tokenizer.39 40## Training Details41 42The model was trained on 1.08 billion tokens for 1 epoch on 64 A100 GPUs, provided by [Stability AI](https://stability.ai/).43 44Each file was formatted as follows for input to the language model:45 46```47<NME> {FILE_NAME}48<BEF> {INPUT_FILE}49<MSG> {COMMIT_MESSAGE}50<DFF> {FILE_DIFF}51```52 53## Intended Uses and Limitations54 55Due to the model’s small size and restriction to code, one should not expect the model to generalize to domains beyond code and perform (successful) reasoning over large chunks of code. This model is intended to be used in prototyping code generation systems, and for solely experimental purposes. This model is provided without warranty and should not be used in commercial settings—even though the license permits.56 57## Limitations and Biases58 59Due to the short context length restriction and due to the fact that all repositories with under 100 stars were excluded, we expect our diff model to underperform on underrepresented languages, for instance Lean or Coq.60 61The output of this model should not be trusted as correct and secure code. This model should not be used in any mission critical setting where security is of importance. When running the output of this model, it should be done as much as possible in a sandbox, such as [gVisor](https://gvisor.dev), since it is very possible for the model to produce code which may delete files, send HTTP requests, or otherwise contain critical security vulnerabilities.62 63As with other language models, diff-codegen is prone to hallucination and biased, stereotyped, or toxic output. There are no guarantees of truthful output when generating from the model.64 65## Evaluation Results66 67See [our blog post](https://carper.ai/diff-model) for full evaluation results.68 69## Licensing70 71This model is licensed as MIT.72 73## Acknowledgements74 75We’d like to thank Honglu Fan, Harry Saini, Herbie Bradley, Reshinth Adithyan, and Joel Lehman for their efforts! Thanks to Nitarshan Rajkumar for feedback on this model card.76