codeparrot/code-generation-models

1The CodeGen architecture follows a standard transformer decoder with left-to-right causal masking. With rotary position embedding for the positional encoding [(Su et al., 2021)](https://arxiv.org/abs/2104.09864), and a context length of 2048. CodeGen models are trained in various sizes. 2 3<div align="center">4 5|Model | # parameters | 6|   -   |   -  | 7| [Salesforce/codegen-350m-mono](https://huggingface.co/Salesforce/codegen-350-mono) | 350M |8| [Salesforce/codegen-2B-mono](https://huggingface.co/Salesforce/codegen-2B-mono) | 2.7B |9| [Salesforce/codegen-6B-mono](https://huggingface.co/Salesforce/codegen-6B-mono) | 6.1B |10| [Salesforce/codegen-16B-mono](https://huggingface.co/Salesforce/codegen-16B-mono) | 16.1B |11 12</div>13 14 15You can load the model and tokenizer directly from 🤗 [`transformers`](https://huggingface.co/docs/transformers/index):16 17```python18from transformers import AutoTokenizer, AutoModelForCausalLM19 20tokenizer = AutoTokenizer.from_pretrained('Salesforce/codegen-16B-mono')21model = AutoModelForCausalLM.from_pretrained('Salesforce/codegen-16B-mono')22 23inputs = tokenizer("def hello_world():", return_tensors="pt")24outputs = model(**inputs)25```