bigcode/astraios-3b-parallel

1 2---3license: bigcode-openrail-m4datasets:5- bigcode/guanaco-commits6metrics:7- code_eval8library_name: peft9tags:10- code11---12# Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models13<p align="center" width="100%">14<a ><img src="https://github.com/bigcode-project/astraios/blob/main/visuals/banner.png?raw=true" alt="Astraios" style="width: 20%; min-width: 300px; display: block; margin: auto;"></a>15</p>16 17# Table of Contents18 191. [Model Summary](#model-summary)202. [Use](#use)213. [Training](#training)224. [Citation](#citation)23 24# Model Summary25 26> Astraios-3B-Parallel Adapter is an instruction tuned model with 15.5B parameters created by finetuning StarCoderBase on CommitPackFT & OASST as described in the Astraios paper.27 28- **Repository:** [bigcode-project/astraios](https://github.com/bigcode-project/astraios)29- **Paper:** [Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models]()30- **Languages:** 80+ Programming languages31- **✨Astraios:**32<table>33<tr>34<th>Data</t> 35<td><a href=https://huggingface.co/datasets/bigcode/guanaco-commits>CommitPackFT+OASST</a></td>36<td>Filtered version of CommitPack and OASST for high-quality commit messages that resemble instructions</td>37</tr>38<tr>39<th>Model</t> 40<td><a href=https://huggingface.co/collections/bigcode/astraios-1b-6576ff1b8e449026ae327c1c>Astraios-1B</a></td>41<td>Collection of StarCoderBase-1B models instruction tuned on CommitPackFT + OASST with different tuning methods</td>42</tr>43<tr>44<th></t> 45<td><a href=https://huggingface.co/collections/bigcode/astraios-3b-6577127317ee44ff547252d3>Astraios-3B</a></td>46<td>Collection of StarCoderBase-3B (3B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods</td>47</tr>48<tr>49<th></t> 50<td><a href=https://huggingface.co/collections/starpeft/starcoderbase-7b-650c1f028b45cfec8e72c265>Astraios-7B</a></td>51<td>Collection of StarCoderBase-7B (7B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods</td>52</tr>53<tr>54<th></t> 55<td><a href=https://huggingface.co/collections/bigcode/astraios-16b-65788b7476b6de79781054cc>Astraios-16B</a></td>56<td>Collection of StarCoderBase-16B (16B parameters) models instruction tuned on CommitPackFT + OASST with different tuning methods</td>57</tr>58<tr>59<th>Evaluation</t> 60<td><a href=https://huggingface.co/datasets/code_x_glue_cc_clone_detection_big_clone_bench>BigCloneBench</a></td>61<td>Dataset for clone detection; We use 2,000 samples for evaluation</td>62</tr>63<tr>64<th></t> 65<td><a href=https://huggingface.co/datasets/code_x_glue_cc_defect_detection>Devign</a></td>66<td>Dataset for defect detection; We use 2,000 samples for evaluation</td>67</tr>68<tr>69<th></t> 70<td><a href=https://huggingface.co/datasets/bigcode/humanevalpack>HumanEvalPack</a></td>71<td>Extension of OpenAI's HumanEval to cover 3 scenarios across 6 languages</td>72</tr>73<tr>74<th></t> 75<td><a href=https://huggingface.co/datasets/RaymondLi/perturbed_humaneval>ReCode</a></td>76<td>Dataset for the robustness of code generation, covering 4 variants</td>77</tr>78<tr>79<th></t> 80<td><a href=https://huggingface.co/datasets/moyix/asleep_keyboard>Asleep At The Keyboard</a></td>81<td>Datasets for security of code generation; We use DoW for evaluation</td>82</tr>83</table>84 85 86# Use87 88## Intended use89 90The model follows instructions provided in the input. You should always preface your input with "Question: " and finish it with "Answer:", for example: "Question: Please write a function in Python that performs bubble sort.91 92Answer:"93 94**Feel free to share your generations in the Community tab!**95 96## Generation97```python98# pip install -q transformers99# pip install -e git+https://github.com/bigcode-project/astraios#subdirectory=peft100from peft import PeftModel 101from transformers import AutoModelForCausalLM, AutoTokenizer102 103peft_checkpoint = "bigcode/astraios-3b-parallel"104checkpoint = "bigcode/starcoderbase-3b"105model = AutoModelForCausalLM.from_pretrained(checkpoint)106model = PeftModel.from_pretrained(model, peft_checkpoint)107device = "cuda" # for GPU usage or "cpu" for CPU usage108 109tokenizer = AutoTokenizer.from_pretrained(checkpoint)110model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)111 112inputs = tokenizer.encode("Question: Please write a function in Python that performs bubble sort.113 114Answer:", return_tensors="pt").to(device)115outputs = model.generate(inputs)116print(tokenizer.decode(outputs[0]))117```118 119# Training120 121## Model122 123- **Architecture:** GPT-2 model with multi-query attention and Fill-in-the-Middle objective124- **Steps:** 250k pretraining & 200 instruction tuning125- **Precision:** fp32126 127## Hardware128 129- **Pretraining:**130  - **GPUs:** 512 Tesla A100131  - **Training time:** 24 days132- **Instruction tuning:**133  - **GPUs:** 8 Tesla A100134 135## Software136 137- **Orchestration:** [Megatron-LM/Transformers](https://github.com/bigcode-project/octopack#training)138- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)139 140# Citation141 142```bibtex143```144