bigcode/santacoder

1---2license: bigcode-openrail-m3datasets:4- bigcode/the-stack5language:6- code7programming_language:8- Java9- JavaScript10- Python11pipeline_tag: text-generation12inference: true13widget:14- text: 'def print_hello_world():'15  example_title: Hello world16  group: Python17model-index:18- name: SantaCoder19  results:20  - task:21      type: text-generation22    dataset:23      type: nuprl/MultiPL-E24      name: MultiPL HumanEval (Python)25    metrics:26    - name: pass@127      type: pass@128      value: 0.1829      verified: false30    - name: pass@1031      type: pass@1032      value: 0.2933      verified: false34    - name: pass@10035      type: pass@10036      value: 0.4937      verified: false38  - task:39      type: text-generation40    dataset:41      type: nuprl/MultiPL-E42      name: MultiPL MBPP (Python)43    metrics:44    - name: pass@145      type: pass@146      value: 0.3547      verified: false48    - name: pass@1049      type: pass@1050      value: 0.5851      verified: false52    - name: pass@10053      type: pass@10054      value: 0.7755      verified: false56  - task:57      type: text-generation58    dataset:59      type: nuprl/MultiPL-E60      name: MultiPL HumanEval (JavaScript)61    metrics:62    - name: pass@163      type: pass@164      value: 0.1665      verified: false66    - name: pass@1067      type: pass@1068      value: 0.2769      verified: false70    - name: pass@10071      type: pass@10072      value: 0.4773      verified: false74  - task:75      type: text-generation76    dataset:77      type: nuprl/MultiPL-E78      name: MultiPL MBPP (Javascript)79    metrics:80    - name: pass@181      type: pass@182      value: 0.2883      verified: false84    - name: pass@1085      type: pass@1086      value: 0.5187      verified: false88    - name: pass@10089      type: pass@10090      value: 0.791      verified: false92  - task:93      type: text-generation94    dataset:95      type: nuprl/MultiPL-E96      name: MultiPL HumanEval (Java)97    metrics:98    - name: pass@199      type: pass@1100      value: 0.15101      verified: false102    - name: pass@10103      type: pass@10104      value: 0.26105      verified: false106    - name: pass@100107      type: pass@100108      value: 0.41109      verified: false110  - task:111      type: text-generation112    dataset:113      type: nuprl/MultiPL-E114      name: MultiPL MBPP (Java)115    metrics:116    - name: pass@1117      type: pass@1118      value: 0.28119      verified: false120    - name: pass@10121      type: pass@10122      value: 0.44123      verified: false124    - name: pass@100125      type: pass@100126      value: 0.59127      verified: false128  - task:129      type: text-generation130    dataset:131      type: loubnabnl/humaneval_infilling132      name: HumanEval FIM (Python)133    metrics:134    - name: single_line135      type: exact_match136      value: 0.44137      verified: false138  - task:139      type: text-generation140    dataset:141      type: nuprl/MultiPL-E142      name: MultiPL HumanEval FIM (Java)143    metrics:144    - name: single_line145      type: exact_match146      value: 0.62147      verified: false148  - task:149      type: text-generation150    dataset:151      type: nuprl/MultiPL-E152      name: MultiPL HumanEval FIM (JavaScript)153    metrics:154    - name: single_line155      type: exact_match156      value: 0.6157      verified: false158  - task:159      type: text-generation160    dataset:161      type: code_x_glue_ct_code_to_text162      name: CodeXGLUE code-to-text (Python)163    metrics:164    - name: BLEU165      type: bleu166      value: 18.13167      verified: false168---169 170# SantaCoder171 172![banner](https://huggingface.co/datasets/bigcode/admin/resolve/main/banner.png)173 174Play with the model on the [SantaCoder Space Demo](https://huggingface.co/spaces/bigcode/santacoder-demo).175 176#  Table of Contents177 1781. [Model Summary](#model-summary)1792. [Use](#use)1803. [Limitations](#limitations)1814. [Training](#training)1825. [License](#license)1836. [Citation](#citation)184 185# Model Summary186 187The SantaCoder models are a series of 1.1B parameter models trained on the Python, Java, and JavaScript subset of [The Stack (v1.1)](https://huggingface.co/datasets/bigcode/the-stack) (which excluded opt-out requests). 188The main model uses [Multi Query Attention](https://arxiv.org/abs/1911.02150), a context window of 2048 tokens, and was trained using near-deduplication and comment-to-code ratio as filtering criteria and using the [Fill-in-the-Middle objective](https://arxiv.org/abs/2207.14255).189In addition there are several models that were trained on datasets with different filter parameters and with architecture and objective variations. 190 191- **Repository:** [bigcode/Megatron-LM](https://github.com/bigcode-project/Megatron-LM)192- **Project Website:** [bigcode-project.org](https://www.bigcode-project.org)193- **Paper:** [🎅SantaCoder: Don't reach for the stars!🌟](https://arxiv.org/abs/2301.03988)194- **Point of Contact:** [contact@bigcode-project.org](mailto:contact@bigcode-project.org)195- **Languages:** Python, Java, and JavaScript196 197|Model|Architecture|Objective|Filtering|198|:-|:-|:-|:-|199|`mha`|MHA|AR + FIM| Base |200|`no-fim`| MQA | AR| Base |201|`fim`| MQA | AR + FIM | Base |202|`stars`| MQA | AR + FIM | GitHub stars |203|`fertility`| MQA | AR + FIM | Tokenizer fertility |204|`comments`| MQA | AR + FIM | Comment-to-code ratio |205|`dedup-alt`| MQA | AR + FIM | Stronger near-deduplication |206|`final`| MQA | AR + FIM | Stronger near-deduplication and comment-to-code ratio |207 208The `final` model is the best performing model and was trained twice as long (236B tokens) as the others. This checkpoint is the default model and available on the `main` branch. All other checkpoints are on separate branches with according names.209 210# Use211 212## Intended use213 214The model was trained on GitHub code. As such it is _not_ an instruction model and commands like "Write a function that computes the square root." do not work well.215You should phrase commands like they occur in source code such as comments (e.g. `# the following function computes the sqrt`) or write a function signature and docstring and let the model complete the function body.216 217**Feel free to share your generations in the Community tab!**218 219## How to use220 221### Generation222```python223# pip install -q transformers224from transformers import AutoModelForCausalLM, AutoTokenizer225 226checkpoint = "bigcode/santacoder"227device = "cuda" # for GPU usage or "cpu" for CPU usage228 229tokenizer = AutoTokenizer.from_pretrained(checkpoint)230model = AutoModelForCausalLM.from_pretrained(checkpoint, trust_remote_code=True).to(device)231 232inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)233outputs = model.generate(inputs)234print(tokenizer.decode(outputs[0]))235```236 237### Fill-in-the-middle238Fill-in-the-middle uses special tokens to identify the prefix/middle/suffix part of the input and output:239 240```python241input_text = "<fim-prefix>def print_hello_world():\n    <fim-suffix>\n    print('Hello world!')<fim-middle>"242inputs = tokenizer.encode(input_text, return_tensors="pt").to(device)243outputs = model.generate(inputs)244print(tokenizer.decode(outputs[0]))245```246Make sure to use `<fim-prefix>, <fim-suffix>, <fim-middle>` and not  `<fim_prefix>, <fim_suffix>, <fim_middle>` as in StarCoder models.247 248### Load other checkpoints249We upload the checkpoint of each experiment to a separate branch as well as the intermediate checkpoints as commits on the branches. You can load them with the `revision` flag:250 251```python252model = AutoModelForCausalLM.from_pretrained(253    "bigcode/santacoder",254    revision="no-fim", # name of branch or commit hash255    trust_remote_code=True256)257```258 259### Attribution & Other Requirements260 261The pretraining dataset of the model was filtered for permissive licenses only. Nevertheless, the model can generate source code verbatim from the dataset. The code's license might require attribution and/or other specific requirements that must be respected. We provide a [search index](https://huggingface.co/spaces/bigcode/santacoder-search) that let's you search through the pretraining data to identify where generated code came from and apply the proper attribution to your code.262 263# Limitations264 265The model has been trained on source code in Python, Java, and JavaScript. The predominant language in source is English although other languages are also present. As such the model is capable to generate code snippets provided some context but the generated code is not guaranteed to work as intended. It can be inefficient, contain bugs or exploits.266 267# Training268 269## Model270 271- **Architecture:** GPT-2 model with multi-query attention and Fill-in-the-Middle objective272- **Pretraining steps:** 600K273- **Pretraining tokens:** 236 billion274- **Precision:** float16275 276## Hardware277 278- **GPUs:** 96 Tesla V100279- **Training time:** 6.2 days280- **Total FLOPS:** 2.1 x 10e21281 282## Software283 284- **Orchestration:** [Megatron-LM](https://github.com/bigcode-project/Megatron-LM)285- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)286- **FP16 if applicable:** [apex](https://github.com/NVIDIA/apex)287 288# License289The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement [here](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement).290 291# Citation292```293@article{allal2023santacoder,294  title={SantaCoder: don't reach for the stars!},295  author={Allal, Loubna Ben and Li, Raymond and Kocetkov, Denis and Mou, Chenghao and Akiki, Christopher and Ferrandis, Carlos Munoz and Muennighoff, Niklas and Mishra, Mayank and Gu, Alex and Dey, Manan and others},296  journal={arXiv preprint arXiv:2301.03988},297  year={2023}298}299```