amanpreetsingh459/llama-2-7b-chat_q4_quantized_cpp

llama-2-7b-chatq4quantized_cpp

• —This model contains the 4-bit quantized version of llama2-7B-chat model in cpp.
• —This can be run on a local cpu system as a cpp module (instructions for the same are given below).
• —As for the testing, the model has been tested on Linux(Ubuntu) os with 12 GB RAM and core i5 processor.
• —The performance is roughly ~3 tokens per second

Usage:

1. 1.Clone the llama C++ repository from github:<br> git clone https://github.com/ggerganov/llama.cpp.git
2. 2.Enter the llama.cpp repository(which was downloaded in the step 1) and build it by running the make command<br> cd llama.cpp <br> make
3. 3.Create a directory names 7B under the directory llama.cpp/models and put the model file ggml-model-q4_0.bin under this newly created 7B directory<br> cd models <br> mkdir 7B
4. 4.Navigate back to llama.cpp directory and run the below command:-<br> ./main -m ./models/7B/ggml-model-q4_0.bin -n 1024 --repeat_penalty 1.0 --color -i -r "User:" -f ./prompts/alpaca.txt <br>

the initial prompt file can be changed to anything from prompts/alpaca.txt to of your choice

1. 1.That's it. Enter the desired prompts and let the results surprise you...

Credits:

1. 1.https://github.com/facebookresearch/llama
2. 2.https://github.com/ggerganov/llama.cpp
3. 3.https://medium.com/@karankakwani/build-and-run-llama2-llm-locally-a3b393c1570e