RichardErkhov/bigcode_-_gpt_bigcode-santacoder-8bits

1Quantization made by Richard Erkhov.2 3[Github](https://github.com/RichardErkhov)4 5[Discord](https://discord.gg/pvy7H8DZMG)6 7[Request more models](https://github.com/RichardErkhov/quant_request)8 9 10gpt_bigcode-santacoder - bnb 8bits11- Model creator: https://huggingface.co/bigcode/12- Original model: https://huggingface.co/bigcode/gpt_bigcode-santacoder/13 14 15 16 17Original model description:18---19license: openrail20datasets:21- bigcode/the-stack22language:23- code24programming_language: 25- Java26- JavaScript27- Python28pipeline_tag: text-generation29inference: false30 31model-index:32- name: SantaCoder33  results:34  - task:35      type: text-generation36    dataset:37      type: nuprl/MultiPL-E38      name: MultiPL HumanEval (Python)39    metrics:40    - name: pass@141      type: pass@142      value: 0.1843      verified: false44    - name: pass@1045      type: pass@1046      value: 0.2947      verified: false48    - name: pass@10049      type: pass@10050      value: 0.4951      verified: false52  - task:53      type: text-generation54    dataset:55      type: nuprl/MultiPL-E56      name: MultiPL MBPP (Python)57    metrics:58    - name: pass@159      type: pass@160      value: 0.3561      verified: false62    - name: pass@1063      type: pass@1064      value: 0.5865      verified: false66    - name: pass@10067      type: pass@10068      value: 0.7769      verified: false70  - task:71      type: text-generation72    dataset:73      type: nuprl/MultiPL-E74      name: MultiPL HumanEval (JavaScript)75    metrics:76    - name: pass@177      type: pass@178      value: 0.1679      verified: false80    - name: pass@1081      type: pass@1082      value: 0.2783      verified: false84    - name: pass@10085      type: pass@10086      value: 0.4787      verified: false88  - task:89      type: text-generation90    dataset:91      type: nuprl/MultiPL-E92      name: MultiPL MBPP (Javascript)93    metrics:94    - name: pass@195      type: pass@196      value: 0.2897      verified: false98    - name: pass@1099      type: pass@10100      value: 0.51101      verified: false102    - name: pass@100103      type: pass@100104      value: 0.70105      verified: false106  - task:107      type: text-generation108    dataset:109      type: nuprl/MultiPL-E110      name: MultiPL HumanEval (Java)111    metrics:112    - name: pass@1113      type: pass@1114      value: 0.15115      verified: false116    - name: pass@10117      type: pass@10118      value: 0.26119      verified: false120    - name: pass@100121      type: pass@100122      value: 0.41123      verified: false124  - task:125      type: text-generation126    dataset:127      type: nuprl/MultiPL-E128      name: MultiPL MBPP (Java)129    metrics:130    - name: pass@1131      type: pass@1132      value: 0.28133      verified: false134    - name: pass@10135      type: pass@10136      value: 0.44137      verified: false138    - name: pass@100139      type: pass@100140      value: 0.59141      verified: false142  - task:143      type: text-generation144    dataset:145      type: loubnabnl/humaneval_infilling146      name: HumanEval FIM (Python)147    metrics:148    - name: single_line149      type: exact_match150      value: 0.44151      verified: false152  - task:153      type: text-generation154    dataset:155      type: nuprl/MultiPL-E156      name: MultiPL HumanEval FIM (Java)157    metrics:158    - name: single_line159      type: exact_match160      value: 0.62161      verified: false162  - task:163      type: text-generation164    dataset:165      type: nuprl/MultiPL-E166      name: MultiPL HumanEval FIM (JavaScript)167    metrics:168    - name: single_line169      type: exact_match170      value: 0.60171      verified: false172  - task:173      type: text-generation174    dataset:175      type: code_x_glue_ct_code_to_text176      name: CodeXGLUE code-to-text (Python)177    metrics:178    - name: BLEU179      type: bleu180      value: 18.13181      verified: false182---183 184# SantaCoder185 186![banner](https://huggingface.co/datasets/bigcode/admin/resolve/main/banner.png)187 188Play with the model on the [SantaCoder Space Demo](https://huggingface.co/spaces/bigcode/santacoder-demo).189 190#  Table of Contents191 1921. [Model Summary](#model-summary)1932. [Use](#use)1943. [Limitations](#limitations)1954. [Training](#training)1965. [License](#license)1976. [Citation](#citation)198 199# Model Summary200 201This is the same model as [SantaCoder](https://huggingface.co/bigcode/santacoder) but it can be loaded with transformers >=4.28.1 to use the GPTBigCode architecture.202We refer the reader to the [SantaCoder model page](https://huggingface.co/bigcode/santacoder) for full documentation about this model203 204 205- **Repository:** [bigcode/Megatron-LM](https://github.com/bigcode-project/Megatron-LM)206- **Project Website:** [bigcode-project.org](www.bigcode-project.org)207- **Paper:** [🎅SantaCoder: Don't reach for the stars!🌟](https://t.co/YV3pzUbYOr)208- **Point of Contact:** [contact@bigcode-project.org](mailto:contact@bigcode-project.org)209- **Languages:** Python, Java, and JavaScript210 211There are two versions (branches) of the model:212* `main`: Uses the `gpt_bigcode` model. [Requires the bigcode fork of transformers](https://github.com/bigcode-project/transformers).213* `main_custom`: Packaged with its modeling code. Requires `transformers>=4.27`.214  Alternatively, it can run on older versions by setting the configuration parameter `activation_function = "gelu_pytorch_tanh"`.215 216# Use217 218## Intended use219 220The model was trained on GitHub code. As such it is _not_ an instruction model and commands like "Write a function that computes the square root." do not work well.221You should phrase commands like they occur in source code such as comments (e.g. `# the following function computes the sqrt`) or write a function signature and docstring and let the model complete the function body.222 223### Attribution & Other Requirements224 225The pretraining dataset of the model was filtered for permissive licenses only. Nevertheless, the model can generate source code verbatim from the dataset. The code's license might require attribution and/or other specific requirements that must be respected. We provide a [search index](https://huggingface.co/spaces/bigcode/santacoder-search) that let's you search through the pretraining data to identify where generated code came from and apply the proper attribution to your code.226 227# Limitations228 229The model has been trained on source code in Python, Java, and JavaScript. The predominant language in source is English although other languages are also present. As such the model is capable to generate code snippets provided some context but the generated code is not guaranteed to work as intended. It can be inefficient, contain bugs or exploits.230 231# Training232 233## Model234 235- **Architecture:** GPT-2 model with multi-query attention and Fill-in-the-Middle objective236- **Pretraining steps:** 600K237- **Pretraining tokens:** 236 billion238- **Precision:** float16239 240## Hardware241 242- **GPUs:** 96 Tesla V100243- **Training time:** 6.2 days244- **Total FLOPS:** 2.1 x 10e21245 246## Software247 248- **Orchestration:** [Megatron-LM](https://github.com/bigcode-project/Megatron-LM)249- **Neural networks:** [PyTorch](https://github.com/pytorch/pytorch)250- **FP16 if applicable:** [apex](https://github.com/NVIDIA/apex)251 252# License253The model is licenses under the CodeML Open RAIL-M v0.1 license. You can find the full license [here](https://huggingface.co/spaces/bigcode/license).254 255 256