facebook/incoder-1B

1---2license: "cc-by-nc-4.0"3tags:4- code5- python6- javascript7---8 9# InCoder 1B10 11A 1B parameter decoder-only Transformer model trained on code using a causal-masked objective, which allows inserting/infilling code as well as standard left-to-right generation.12 13The model was trained on public open-source repositories with a permissive, non-copyleft, license (Apache 2.0, MIT, BSD-2 or BSD-3) from GitHub and GitLab, as well as StackOverflow. Repositories primarily contained Python and JavaScript, but also include code from 28 languages, as well as StackOverflow. 14 15For more information, see our:16 17- [Demo](https://huggingface.co/spaces/facebook/incoder-demo)18- [Project site](https://sites.google.com/view/incoder-code-models)19- [Examples](https://sites.google.com/view/incoder-code-models/home/examples)20- [Paper](https://arxiv.org/abs/2204.05999)21 22A larger, 6B, parameter model is also available at [facebook/incoder-6B](https://huggingface.co/facebook/incoder-6B).23 24## Requirements25 26`pytorch`, `tokenizers`, and `transformers`. Our model requires HF's tokenizers >= 0.12.1, due to changes in the pretokenizer.27 28```29pip install torch30pip install "tokenizers>=0.12.1"31pip install transformers32```33 34## Usage35 36See [https://github.com/dpfried/incoder](https://github.com/dpfried/incoder) for example code.37 38### Model39`model = AutoModelForCausalLM.from_pretrained("facebook/incoder-1B")`40 41### Tokenizer42`tokenizer = AutoTokenizer.from_pretrained("facebook/incoder-1B")`43 44(Note: the incoder-1B and incoder-6B tokenizers are identical, so 'facebook/incoder-6B' could also be used.)45 46When calling `tokenizer.decode`, it's important to pass `clean_up_tokenization_spaces=False` to avoid removing spaces after punctuation. For example:47 48`tokenizer.decode(tokenizer.encode("from ."), clean_up_tokenization_spaces=False)`49 50(Note: encoding prepends the `<|endoftext|>` token, as this marks the start of a document to our model. This token can be removed from the decoded output by passing `skip_special_tokens=True` to `tokenizer.decode`.)51 52## License53 54CC-BY-NC 4.055 56## Credits57 58The model was developed by Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer and Mike Lewis.59 60Thanks to Lucile Saulnier, Leandro von Werra, Nicolas Patry, Suraj Patil, Omar Sanseviero, and others at HuggingFace for help with the model release, and to Naman Goyal and Stephen Roller for the code our demo was based on!