QuantFactory/starcoder2-7b-instruct-GGUF
21.6k
1 2---3 4tags:5- code6- starcoder27library_name: transformers8pipeline_tag: text-generation9license: bigcode-openrail-m10 11---12 13[](https://hf.co/QuantFactory)14 15 16# QuantFactory/starcoder2-7b-instruct-GGUF17This is quantized version of [TechxGenus/starcoder2-7b-instruct](https://huggingface.co/TechxGenus/starcoder2-7b-instruct) created using llama.cpp18 19# Original Model Card20 21 22<p align="center">23<img width="300px" alt="starcoder2-instruct" src="https://huggingface.co/TechxGenus/starcoder2-7b-instruct/resolve/main/starcoder2-instruct.jpg">24</p>25 26### starcoder2-instruct27 28We've fine-tuned starcoder2-7b with an additional 0.7 billion high-quality, code-related tokens for 3 epochs. We used DeepSpeed ZeRO 3 and Flash Attention 2 to accelerate the training process. It achieves **73.2 pass@1** on HumanEval-Python. This model operates using the Alpaca instruction format (excluding the system prompt).29 30### Usage31 32Here give some examples of how to use our model:33 34```python35from transformers import AutoTokenizer, AutoModelForCausalLM36import torch37PROMPT = """### Instruction38{instruction}39### Response40"""41instruction = <Your code instruction here>42prompt = PROMPT.format(instruction=instruction)43tokenizer = AutoTokenizer.from_pretrained("TechxGenus/starcoder2-7b-instruct")44model = AutoModelForCausalLM.from_pretrained(45 "TechxGenus/starcoder2-7b-instruct",46 torch_dtype=torch.bfloat16,47 device_map="auto",48)49inputs = tokenizer.encode(prompt, return_tensors="pt")50outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=2048)51print(tokenizer.decode(outputs[0]))52```53 54With text-generation pipeline:55 56 57```python58from transformers import pipeline59import torch60PROMPT = """### Instruction61{instruction}62### Response63"""64instruction = <Your code instruction here>65prompt = PROMPT.format(instruction=instruction)66generator = pipeline(67 model="TechxGenus/starcoder2-7b-instruct",68 task="text-generation",69 torch_dtype=torch.bfloat16,70 device_map="auto",71)72result = generator(prompt, max_length=2048)73print(result[0]["generated_text"])74```75 76### Note77 78Model may sometimes make errors, produce misleading contents, or struggle to manage tasks that are not related to coding. It has undergone very limited testing. Additional safety testing should be performed before any real-world deployments.79 80