stabilityai/stable-code-3b
6687.7k
1---2license: other3datasets:4- tiiuae/falcon-refinedweb5- bigcode/the-stack-github-issues6- bigcode/commitpackft7- bigcode/starcoderdata8- EleutherAI/proof-pile-29- meta-math/MetaMathQA10language:11- en12tags:13- causal-lm14- code15metrics:16- code_eval17library_name: transformers18model-index:19- name: stabilityai/stable-code-3b20 results:21 - task:22 type: text-generation23 dataset:24 type: nuprl/MultiPL-E25 name: MultiPL-HumanEval (Python)26 metrics:27 - name: pass@128 type: pass@129 value: 32.430 verified: false31 - task:32 type: text-generation33 dataset:34 type: nuprl/MultiPL-E35 name: MultiPL-HumanEval (C++)36 metrics:37 - name: pass@138 type: pass@139 value: 30.940 verified: false41 - task:42 type: text-generation43 dataset:44 type: nuprl/MultiPL-E45 name: MultiPL-HumanEval (Java)46 metrics:47 - name: pass@148 type: pass@149 value: 32.150 verified: false51 - task:52 type: text-generation53 dataset:54 type: nuprl/MultiPL-E55 name: MultiPL-HumanEval (JavaScript)56 metrics:57 - name: pass@158 type: pass@159 value: 32.160 verified: false61 - task:62 type: text-generation63 dataset:64 type: nuprl/MultiPL-E65 name: MultiPL-HumanEval (PHP)66 metrics:67 - name: pass@168 type: pass@169 value: 24.270 verified: false71 - task:72 type: text-generation73 dataset:74 type: nuprl/MultiPL-E75 name: MultiPL-HumanEval (Rust)76 metrics:77 - name: pass@178 type: pass@179 value: 23.080 verified: false81---82# `stable-code-3b`83 84Please note: For commercial use, please refer to https://stability.ai/license.85 86## Model Description87 88`stable-code-3b` is a 2.7B billion parameter decoder-only language model pre-trained on 1.3 trillion tokens of diverse textual and code datasets. `stable-code-3b` is trained on 18 programming languages (selected based on the 2023 StackOverflow Developer Survey) and demonstrates state-of-the-art performance (compared to models of similar size) on the MultiPL-E metrics across multiple programming languages tested using [BigCode's Evaluation Harness](https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main).89 9091 92| Model | Size | Python | C++ | Javascript | Java | PHP | Rust |93|------------------|------|--------|------|------------|------|------|------|94| **Stable Code** | 3B | 32.4% | 30.9%| 32.1% | 32.1%| 24.2%| 23.0%|95| CodeLLama | 7B | 30.0% | 28.2%| 32.5% | 31.1%| 25.7%| 26.3%|96| Deepseek Coder | 1.3B | 28.6% | 29.2%| 28.7% | 29.0%| 23.6%| 18.5%|97| Wizard Coder | 3B | 31.6% | 25.6%| 26.2% | 25.8%| 25.3%| 20.4%|98| StarCoder | 3B | 21.6% | 19.8%| 21.5% | 20.5%| 19.0%| 16.9%|99| Replit Code V1.5 | 3B | 23.0% | 25.9%| 26.2% | 23.6%| 23.2%| 21.5%|100| Deci Coder | 1B | 19.1% | 6.8% | 18.4% | 16.7%| 2.1% | 1.7% |101 102**Key Features**103* Fill in Middle Capability (FIM)104* Supports Long Context, trained with Sequences upto 16,384105 106## Usage107 108Get started generating text with `stable-code-3b` by using the following code snippet:109 110```python111import torch112from transformers import AutoModelForCausalLM, AutoTokenizer113tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-3b")114model = AutoModelForCausalLM.from_pretrained(115 "stabilityai/stable-code-3b",116 torch_dtype="auto",117)118model.cuda()119inputs = tokenizer("import torch\nimport torch.nn as nn", return_tensors="pt").to(model.device)120tokens = model.generate(121 **inputs,122 max_new_tokens=48,123 temperature=0.2,124 do_sample=True,125)126print(tokenizer.decode(tokens[0], skip_special_tokens=True))127```128 129### Run with Fill in Middle (FIM) ⚡️130 131<details>132<summary> Click to expand </summary>133 134```python135from transformers import AutoModelForCausalLM, AutoTokenizer136tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-3b")137model = AutoModelForCausalLM.from_pretrained(138 "stabilityai/stable-code-3b",139 torch_dtype="auto",140 attn_implementation="flash_attention_2",141)142model.cuda()143inputs = tokenizer("<fim_prefix>def fib(n):<fim_suffix> else:\n return fib(n - 2) + fib(n - 1)<fim_middle>", return_tensors="pt").to(model.device)144tokens = model.generate(145 **inputs,146 max_new_tokens=48,147 temperature=0.2,148 do_sample=True,149)150print(tokenizer.decode(tokens[0], skip_special_tokens=True))151```152 153</details>154 155### Run with Flash Attention 2 ⚡️156 157<details>158<summary> Click to expand </summary>159 160```python161from transformers import AutoModelForCausalLM, AutoTokenizer162tokenizer = AutoTokenizer.from_pretrained("stabilityai/stable-code-3b", trust_remote_code=True)163model = AutoModelForCausalLM.from_pretrained(164 "stabilityai/stable-code-3b",165 trust_remote_code=True,166 torch_dtype="auto",167+ attn_implementation="flash_attention_2",168)169model.cuda()170inputs = tokenizer("import torch\nimport torch.nn as nn", return_tensors="pt").to(model.device)171tokens = model.generate(172 **inputs,173 max_new_tokens=48,174 temperature=0.2,175 do_sample=True,176)177print(tokenizer.decode(tokens[0], skip_special_tokens=True))178```179 180</details>181 182 183## Model Details184 185* **Developed by**: [Stability AI](https://stability.ai/)186* **Model type**: `stable-code-3b` models are auto-regressive language models based on the transformer decoder architecture.187* **Language(s)**: English, Code188* **Library**: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)189* **License**: Stability AI Community License.190* **Commercial License**: to use this model commercially, please refer to https://stability.ai/license191* **Contact**: For questions and comments about the model, please email `lm@stability.ai`192 193### Model Architecture194 195The model is a decoder-only transformer similar to the LLaMA ([Touvron et al., 2023](https://arxiv.org/abs/2307.09288)) architecture with the following modifications:196 197| Parameters | Hidden Size | Layers | Heads | Sequence Length |198|----------------|-------------|--------|-------|-----------------|199| 2,796,431,360 | 2560 | 32 | 32 | 16384 |200 201* **Position Embeddings**: Rotary Position Embeddings ([Su et al., 2021](https://arxiv.org/abs/2104.09864)) applied to the first 25% of head embedding dimensions for improved throughput following [Black et al. (2022)](https://arxiv.org/pdf/2204.06745.pdf).202* **Tokenizer**: We use a modified version of the GPTNeoX Tokenizer.[`NeoX`](https://github.com/EleutherAI/gpt-neox). We add special tokens to train for Fill in the Middle (FIM) capabilities like `<FIM_PREFIX>` and `<FIM_SUFFIX>` along with other special tokens.203 204## Training205 206### Training Dataset207 208The dataset is comprised of a filtered mixture of open-source large-scale datasets available on the [HuggingFace Hub](https://huggingface.co/datasets): Falcon RefinedWeb extract ([Penedo et al., 2023](https://huggingface.co/datasets/tiiuae/falcon-refinedweb)), along with [CommitPackFT](https://huggingface.co/datasets/bigcode/commitpackft) and [Github Issues](https://huggingface.co/datasets/bigcode/the-stack-github-issues) (BigCode., 2023), and StarCoder ([Li et al., 2023](https://arxiv.org/abs/2305.06161)). We further supplement our training with data from mathematical domains ([Azerbayev, Zhangir, et al., 2023](https://arxiv.org/abs/2310.10631) and, [Yu, Longhui, et al., 2023](https://arxiv.org/abs/2309.12284)). 209 210Top 18 programming languages trained on:211- C212- CPP213- Java214- JavaScript215- CSS216- Go217- HTML218- Ruby219- Rust220- Markdown221- Shell222- Php223- Sql224- R225- Typescript226- Python227- Jupyter-Clean228- RestructuredText229 230### Training Procedure231 232The model is pre-trained on the aforementioned datasets in `bfloat16` precision, optimized with AdamW.233 234### Training Infrastructure235 236* **Hardware**: `stable-code-3b` was trained on the Stability AI cluster across 256 NVIDIA A100 40GB GPUs (AWS P4d instances).237 238* **Software**: We use a fork of `gpt-neox` ([EleutherAI, 2021](https://github.com/EleutherAI/gpt-neox)), train under 2D parallelism (Data and Tensor Parallel) with ZeRO-1 ([Rajbhandari et al., 2019](https://arxiv.org/abs/1910.02054v3)), and rely on flash-attention as well as SwiGLU and Rotary Embedding kernels from FlashAttention-2 ([Dao et al., 2023](https://tridao.me/publications/flash2/flash2.pdf))239 240## Use and Limitations241 242### Intended Use243 244The model is intended to be used as a foundational base model for application-specific fine-tuning. Developers must evaluate and fine-tune the model for safe performance in downstream applications. For commercial use, please refer to https://stability.ai/license.245 246### Limitations and Bias247248As a base model, this model may exhibit unreliable, unsafe, or other undesirable behaviors that must be corrected through evaluation and fine-tuning prior to deployment. The pre-training dataset may have contained offensive or inappropriate content, even after applying data cleansing filters, which can be reflected in the model-generated text. We recommend that users exercise caution when using these models in production systems. Do not use the models if they are unsuitable for your application, or for any applications that may cause deliberate or unintentional harm to others.249 250## How to Cite251 252```bibtex253@misc{stable-code-3b,254 url={[https://huggingface.co/stabilityai/stable-code-3b](https://huggingface.co/stabilityai/stable-code-3b)},255 title={Stable Code 3B},256 author={Pinnaparaju, Nikhil and Adithyan, Reshinth and Phung, Duy and Tow, Jonathan and Baicoianu, James and Cooper, Nathan}257}258```