Harvest577/replit-code-v1-3b

1---2license: cc-by-sa-4.03datasets:4- bigcode/the-stack-dedup5tags:6- code7language:8- code9programming_language: 10- Markdown11- Java12- JavaScript13- Python14- TypeScript15- PHP16- SQL17- JSX18- reStructuredText19- Rust20- C21- CSS22- Go23- C++24- HTML25- Vue26- Ruby27- Jupyter Notebook28- R29- Shell30model-index:31- name: replit-code-v1-3b32  results:33  - task: 34      name: Code Generation35      type: code-generation36    dataset:37      name: "HumanEval" 38      type: openai_humaneval39    metrics:40    - name: pass@141      type: pass@142      value: 0.21943      verified: false44---45 46 47# replit-code-v1-3b48Developed by: Replit, Inc.49 50[**🧑‍💻 Test it on our Demo Space! 🧑‍💻**](https://huggingface.co/spaces/replit/replit-code-v1-3b-demo)51 52[**⚙️ Fine-tuning and Instruct-tuning guides ⚙️**](https://github.com/replit/replitLM)53 54## Model Description55`replit-code-v1-3b` is a 2.7B Causal Language Model focused on **Code Completion**. The model has been trained on a subset of the [Stack Dedup v1.2 dataset](https://arxiv.org/abs/2211.15533).56 57The training mixture includes **20 different languages**, listed here in descending order of number of tokens: 58<br/>59`Markdown`, `Java`, `JavaScript`, `Python`, `TypeScript`, `PHP`, `SQL`, `JSX`, `reStructuredText`, `Rust`, `C`, `CSS`, `Go`, `C++`, `HTML`, `Vue`, `Ruby`, `Jupyter Notebook`, `R`, `Shell`60<br/>61In total, the training dataset contains 175B tokens, which were repeated over 3 epochs -- in total, `replit-code-v1-3b` has been trained on **525B** tokens (~195 tokens per parameter).62 63The model has been trained on the [MosaicML](https://www.mosaicml.com/) platform with 256 x A100-40GB GPUs, leveraging their latest [LLM examples repo](https://github.com/mosaicml/examples/tree/release/v0.0.4/examples/llm).64<br/>65`replit-code-v1-3b` is powered by state-of-the-art LLM techniques, such as: 66[Flash Attention](https://arxiv.org/abs/2205.14135) for fast training and inference,67[AliBi positional embeddings](https://arxiv.org/abs/2108.12409) to support variable context length at inference time, 68[LionW optimizer](https://arxiv.org/abs/2302.06675), 69etc.70 71## Intended Use72Replit intends this model be used by anyone as a foundational model for application-specific fine-tuning without strict limitations on commercial use.73 74## Limitations75The pre-training dataset may have contained offensive or inappropriate content even after applying data cleansing filters, and such content may be reflected in model generated text. We recommend that users exercise reasonable caution when using in production systems. Do not use for any applications that may cause harm or distress to individuals or groups.76 77## License78The model checkpoint and vocabulary file are licensed under the Creative Commons license (CC BY-SA-4.0). Under the license, you must give credit to Replit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests that Replit endorses you or your use.79 80The source code files (`*.py`) are licensed under the Apache 2.0 license.81 82## Contact83For questions and comments about the model, please post in the community section. 84 85## How to Use86First of all, you need to install the latest versions of the following dependencies:87```88einops89sentencepiece90torch91transformers92```93 94You can then load the model as follows:95```python96from transformers import AutoModelForCausalLM97 98# load model99model = AutoModelForCausalLM.from_pretrained('replit/replit-code-v1-3b', trust_remote_code=True)100```101 102To use the optimized Triton implementation of FlashAttention on GPUs with BF16 precision, first install the following dependencies: 103```104flash-attn==0.2.8105triton==2.0.0.dev20221202106```107 108Then, move the model to `bfloat16` and use it as follows:109```python110from transformers import AutoModelForCausalLM, AutoConfig111 112config = AutoConfig.from_pretrained(113    "replit/replit-code-v1-3b",114    trust_remote_code=True115)116config.attn_config['attn_impl'] = 'triton'117 118# load model119model = AutoModelForCausalLM.from_pretrained('replit/replit-code-v1-3b', config=config, trust_remote_code=True)120model.to(device='cuda:0', dtype=torch.bfloat16)121 122# forward pass123x = torch.tensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])124x = x.to(device='cuda:0')125y = model(x)126 127```128 129Note that `trust_remote_code=True` is passed to the `from_pretrained` method because ReplitLM is not a class in the130[Transformers](https://huggingface.co/docs/transformers/index) library. 131 132### Tokenizer133 134We have trained a custom SentencePiece Unigram tokenizer optimized with a vocabulary specifically for code of 32768 tokens.135 136Note that using this requires the `sentencepiece` library to be installed. 137 138The tokenizer can be used as follows:139 140```python141from transformers import AutoTokenizer142 143# load tokenizer144tokenizer = AutoTokenizer.from_pretrained('replit/replit-code-v1-3b', trust_remote_code=True)145 146# single input encoding + generation147x = tokenizer.encode('def hello():\n  print("hello world")\n', return_tensors='pt')148y = model.generate(x)149 150# decoding, clean_up_tokenization_spaces=False to ensure syntactical correctness151generated_code = tokenizer.decode(y[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)152print(generated_code)153```154 155Note that: 156- `trust_remote_code=True` is passed to the `from_pretrained` method because ReplitLM is not a class in the [Transformers](https://huggingface.co/docs/transformers/index) library. 157- `clean_up_tokenization_spaces=False` is meant to avoid removing spaces in the output, because that would affect the syntactical correctness of the generated code. 158 159 160### Generation161 162You can generate code using the `transformers` library as follows:163 164```python165from transformers import AutoModelForCausalLM, AutoTokenizer166 167tokenizer = AutoTokenizer.from_pretrained('replit/replit-code-v1-3b', trust_remote_code=True)168model = AutoModelForCausalLM.from_pretrained('replit/replit-code-v1-3b', trust_remote_code=True)169 170x = tokenizer.encode('def fibonacci(n): ', return_tensors='pt')171y = model.generate(x, max_length=100, do_sample=True, top_p=0.95, top_k=4, temperature=0.2, num_return_sequences=1, eos_token_id=tokenizer.eos_token_id)172 173# decoding, clean_up_tokenization_spaces=False to ensure syntactical correctness174generated_code = tokenizer.decode(y[0], skip_special_tokens=True, clean_up_tokenization_spaces=False)175print(generated_code)176```177 178Experiment with different decoding methods and parameters to get the best results for your use case.179 180 181### Loading with 8-bit and 4-bit quantization182 183#### Loading in 8-bit184You can also load the model in 8-bit with the `load_in_8bit=True` kwarg that uses `bitsandbytes` under the hood.185 186First you need to  install the following additional dependanices: 187```188accelerate189bitsandbytes190```191 192Then you can load the model in 8bit as follows:193 194```195model = AutoModelForCausalLM.from_pretrained("replit/replit-code-v1-3b", 196                                             trust_remote_code=True, 197                                             device_map="auto",198                                             load_in_8bit=True)199```200The additional kwargs that make this possible are `device_map='auto'` and `load_in_8bit=True`. 201 202#### Loading in 4-bit203 204For loading in 4-bit, at the time of writing, support for `load_in_4bit` has not been merged into the latest releases for 205`transformers` and `accelerate`. However you can use it if you install the dependancies the `main` branches of the published repos:206 207```bash208pip install git+https://github.com/huggingface/accelerate.git209pip install git+https://github.com/huggingface/transformers.git210```211 212Then load in 4-bit with:213 214```215model = AutoModelForCausalLM.from_pretrained("replit/replit-code-v1-3b", 216                                             trust_remote_code=True, 217                                             device_map="auto",218                                             load_in_4bit=True)219```220 221#### References222- [Hugging Face's Quantization Doc](https://huggingface.co/docs/transformers/main/main_classes/quantization)223- [Original Blogpost introducing 8-bit](https://huggingface.co/blog/hf-bitsandbytes-integration)224- [New Blogpost introducing 4-bit](https://huggingface.co/blog/4bit-transformers-bitsandbytes)225 226 227### Post Processing228 229Note that as with all code generation models, post-processing of the generated code is important. In particular, the following post-processing steps are recommended:230- stop generation when the EOS token is encountered231- remove trailing whitespaces232- set `max_tokens` to a reasonable value based on your completion use case233- truncate generation to stop words such as `return`, `def`, "```", "`\n\n\n`" to avoid generating incomplete code when `max_tokens` is larger than the length of the expected generated code.