bigcode/octocoder
71664
1---2pipeline_tag: text-generation3inference: true4widget:5- text: 'Question: Please write a function in Python that performs bubble sort.\n\nAnswer:'6 example_title: Bubble sort7 group: Python8license: bigcode-openrail-m9datasets:10- bigcode/commitpackft11- bigcode/oasst-octopack12metrics:13- code_eval14library_name: transformers15tags:16- code17model-index:18- name: OctoCoder19 results:20 - task:21 type: text-generation22 dataset:23 type: bigcode/humanevalpack24 name: HumanEvalSynthesize Python25 metrics:26 - name: pass@127 type: pass@128 value: 46.229 verified: false30 - task:31 type: text-generation32 dataset:33 type: bigcode/humanevalpack34 name: HumanEvalSynthesize JavaScript35 metrics:36 - name: pass@137 type: pass@138 value: 39.239 verified: false40 - task:41 type: text-generation42 dataset:43 type: bigcode/humanevalpack44 name: HumanEvalSynthesize Java45 metrics:46 - name: pass@147 type: pass@148 value: 38.249 verified: false50 - task:51 type: text-generation52 dataset:53 type: bigcode/humanevalpack54 name: HumanEvalSynthesize Go55 metrics:56 - name: pass@157 type: pass@158 value: 30.459 verified: false60 - task:61 type: text-generation62 dataset:63 type: bigcode/humanevalpack64 name: HumanEvalSynthesize C++65 metrics:66 - name: pass@167 type: pass@168 value: 35.669 verified: false70 - task:71 type: text-generation72 dataset:73 type: bigcode/humanevalpack74 name: HumanEvalSynthesize Rust75 metrics:76 - name: pass@177 type: pass@178 value: 23.479 verified: false80 - task:81 type: text-generation82 dataset:83 type: bigcode/humanevalpack84 name: HumanEvalSynthesize Average85 metrics:86 - name: pass@187 type: pass@188 value: 35.589 verified: false90 - task:91 type: text-generation92 dataset:93 type: bigcode/humanevalpack94 name: HumanEvalFix Python95 metrics:96 - name: pass@197 type: pass@198 value: 30.499 verified: false100 - task:101 type: text-generation102 dataset:103 type: bigcode/humanevalpack104 name: HumanEvalFix JavaScript105 metrics:106 - name: pass@1107 type: pass@1108 value: 28.4109 verified: false110 - task:111 type: text-generation112 dataset:113 type: bigcode/humanevalpack114 name: HumanEvalFix Java115 metrics:116 - name: pass@1117 type: pass@1118 value: 30.6119 verified: false120 - task:121 type: text-generation122 dataset:123 type: bigcode/humanevalpack124 name: HumanEvalFix Go125 metrics:126 - name: pass@1127 type: pass@1128 value: 30.2129 verified: false130 - task:131 type: text-generation132 dataset:133 type: bigcode/humanevalpack134 name: HumanEvalFix C++135 metrics:136 - name: pass@1137 type: pass@1138 value: 26.1139 verified: false140 - task:141 type: text-generation142 dataset:143 type: bigcode/humanevalpack144 name: HumanEvalFix Rust145 metrics:146 - name: pass@1147 type: pass@1148 value: 16.5149 verified: false150 - task:151 type: text-generation152 dataset:153 type: bigcode/humanevalpack154 name: HumanEvalFix Average155 metrics:156 - name: pass@1157 type: pass@1158 value: 27.0159 verified: false160 - task:161 type: text-generation162 dataset:163 type: bigcode/humanevalpack164 name: HumanEvalExplain Python165 metrics:166 - name: pass@1167 type: pass@1168 value: 35.1169 verified: false170 - task:171 type: text-generation172 dataset:173 type: bigcode/humanevalpack174 name: HumanEvalExplain JavaScript175 metrics:176 - name: pass@1177 type: pass@1178 value: 24.5179 verified: false180 - task:181 type: text-generation182 dataset:183 type: bigcode/humanevalpack184 name: HumanEvalExplain Java185 metrics:186 - name: pass@1187 type: pass@1188 value: 27.3189 verified: false190 - task:191 type: text-generation192 dataset:193 type: bigcode/humanevalpack194 name: HumanEvalExplain Go195 metrics:196 - name: pass@1197 type: pass@1198 value: 21.1199 verified: false200 - task:201 type: text-generation202 dataset:203 type: bigcode/humanevalpack204 name: HumanEvalExplain C++205 metrics:206 - name: pass@1207 type: pass@1208 value: 24.1209 verified: false210 - task:211 type: text-generation212 dataset:213 type: bigcode/humanevalpack214 name: HumanEvalExplain Rust215 metrics:216 - name: pass@1217 type: pass@1218 value: 14.8219 verified: false220 - task:221 type: text-generation222 dataset:223 type: bigcode/humanevalpack224 name: HumanEvalExplain Average225 metrics:226 - name: pass@1227 type: pass@1228 value: 24.5229 verified: false230---231 232233 234# Table of Contents235 2361. [Model Summary](#model-summary)2372. [Use](#use)2383. [Training](#training)2394. [Citation](#citation)240 241# Model Summary242 243> OctoCoder is an instruction tuned model with 15.5B parameters created by finetuning StarCoder on CommitPackFT & OASST as described in the OctoPack paper.244 245- **Repository:** [bigcode-project/octopack](https://github.com/bigcode-project/octopack)246- **Paper:** [OctoPack: Instruction Tuning Code Large Language Models](https://arxiv.org/abs/2308.07124)247- **Languages:** 80+ Programming languages248- **OctoPack🐙🎒:**249<table>250<tr>251<th>Data</t> 252<th><a href=https://huggingface.co/datasets/bigcode/commitpack>CommitPack</a></th>253<td>4TB of GitHub commits across 350 programming languages</td>254</tr>255<tr>256<th></t> 257<th><a href=https://huggingface.co/datasets/bigcode/commitpackft>CommitPackFT</a></th>258<td>Filtered version of CommitPack for high-quality commit messages that resemble instructions</td>259</tr>260<tr>261<th>Model</t> 262<th><a href=https://huggingface.co/bigcode/octocoder>OctoCoder</a></th>263<td>StarCoder (16B parameters) instruction tuned on CommitPackFT + OASST</td>264</tr>265<tr>266<th></t> 267<th><a href=https://huggingface.co/bigcode/octogeex>OctoGeeX</a></th>268<td>CodeGeeX2 (6B parameters) instruction tuned on CommitPackFT + OASST</td>269</tr>270<tr>271<th>Evaluation </t> 272<th><a href=https://huggingface.co/datasets/bigcode/humanevalpack>HumanEvalPack</a></th>273<td>Extension of OpenAI's HumanEval to cover 3 scenarios across 6 languages</td>274</tr>275</table>276 277 278# Use279 280## Intended use281 282The model follows instructions provided in the input. You should always preface your input with "Question: " and finish it with "Answer:", for example: "Question: Please write a function in Python that performs bubble sort.\n\nAnswer:"283 284**Feel free to share your generations in the Community tab!**285 286## Generation287```python288# pip install -q transformers289from transformers import AutoModelForCausalLM, AutoTokenizer290 291checkpoint = "bigcode/octocoder"292device = "cuda" # for GPU usage or "cpu" for CPU usage293 294tokenizer = AutoTokenizer.from_pretrained(checkpoint)295model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)296 297inputs = tokenizer.encode("Question: Please write a function in Python that performs bubble sort.\n\nAnswer:", return_tensors="pt").to(device)298outputs = model.generate(inputs)299print(tokenizer.decode(outputs[0]))300```301 302# Training303 304## Model305 306- **Architecture:** GPT-2 model with multi-query attention and Fill-in-the-Middle objective307- **Steps:** 250k pretraining & 30 instruction tuning308- **Pretraining tokens:** 1 trillion pretraining & 2M instruction tuning309- **Precision:** bfloat16310 311## Hardware312 313- **Pretraining:**314 - **GPUs:** 512 Tesla A100315 - **Training time:** 24 days316- **Instruction tuning:**317 - **GPUs:** 8 Tesla A100318 - **Training time:** 4 hours319 320## Software321 322- **Orchestration:** [Megatron-LM/Transformers](https://github.com/bigcode-project/octopack#training)323- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)324 325# Citation326 327```bibtex328@article{muennighoff2023octopack,329 title={OctoPack: Instruction Tuning Code Large Language Models}, 330 author={Niklas Muennighoff and Qian Liu and Armel Zebaze and Qinkai Zheng and Binyuan Hui and Terry Yue Zhuo and Swayam Singh and Xiangru Tang and Leandro von Werra and Shayne Longpre},331 journal={arXiv preprint arXiv:2308.07124},332 year={2023}333}334```