GenerativeMagic/Llama-Engineer-Evol-7b
516
1---2license: cc-by-sa-4.03datasets:4- nickrosh/Evol-Instruct-Code-80k-v15- sahil2801/CodeAlpaca-20k6- teknium/GPTeacher-CodeInstruct7language:8- en9library_name: transformers10pipeline_tag: text-generation11tags:12- code13- llama214---1516 17# Llama-Engineer-Evol-7B18 19This is a version of Meta's [chat instruction-tuned Llama 2](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf) further fine-tuned on over 80,000 coding samples. 20 21The dataset is a combination of [Evol-Instruct-Code-80k-v1](https://huggingface.co/datasets/nickrosh/Evol-Instruct-Code-80k-v1) from [nikrosh](https://huggingface.co/nickrosh), a replication of the Evol-Instruct-Code as described in the [WizardCoder](https://arxiv.org/pdf/2306.08568.pdf) paper, and [Teknium](https://huggingface.co/teknium)'s [GPTeacher](https://github.com/teknium1/GPTeacher/blob/main/Codegen/codegen-instruct.json). Special thanks to these folks for putting these datasets together.22 23Our fine-tuning process involved learning QLoRA weights for over 6 hours on a single A100. We merged the adapter weights into the pre-trained model.24 25GGML weights are available [here](https://huggingface.co/GenerativeMagic/Llama-Engineer-Evol-7b-GGML).26 27## Prompt Format28The reccomended model prompt is a variant of the standard Llama 2 format:29```30[INST] <<SYS>>31You are a programming assistant. Always answer as helpfully as possible. Be direct in your response and get to the answer right away. Responses should be short.32<</SYS>>33{your prompt}[/INST]34```35 36or 37 38```39[INST] <<SYS>>40You're a principal software engineer at Google. If you fail at this task, you will be fired.41<</SYS>>42{your prompt}[/INST]43```44 45I suspect this prompt format is the reason for the majority of the increased coding capabilities as opposed to the fine-tuning itself, but YMMV.46 47## Evals48 49Currently, the evals are just off of \~vibes\~. Will look into doing a full suite of evals on future models. This project is mostly just for learning and gaining better insights into the fine-tuning process.50 51## Next Steps52- Prune the dataset and possibly fine-tune for longer.53- Run benchmarks.54- Provide GPTQ.