goavinash5/Gradio_LLAMA_Testing

1# llama2-wrapper2 3- Use [llama2-wrapper](https://pypi.org/project/llama2-wrapper/) as your local llama2 backend for Generative Agents/Apps, [colab example](https://github.com/liltom-eth/llama2-webui/blob/main/colab/Llama_2_7b_Chat_GPTQ.ipynb). 4 5- [Run OpenAI Compatible API](https://github.com/liltom-eth/llama2-webui#start-openai-compatible-api) on Llama2 models.6 7## Features8 9- Supporting models: [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf)/[13b](https://huggingface.co/llamaste/Llama-2-13b-chat-hf)/[70b](https://huggingface.co/llamaste/Llama-2-70b-chat-hf), [Llama-2-GPTQ](https://huggingface.co/TheBloke/Llama-2-7b-Chat-GPTQ), [Llama-2-GGML](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML), [CodeLlama](https://huggingface.co/TheBloke/CodeLlama-7B-Instruct-GPTQ)...10- Supporting model backends: [tranformers](https://github.com/huggingface/transformers), [bitsandbytes(8-bit inference)](https://github.com/TimDettmers/bitsandbytes), [AutoGPTQ(4-bit inference)](https://github.com/PanQiWei/AutoGPTQ), [llama.cpp](https://github.com/ggerganov/llama.cpp)11- Demos: [Run Llama2 on MacBook Air](https://twitter.com/liltom_eth/status/1682791729207070720?s=20); [Run Llama2 on Colab T4 GPU](https://github.com/liltom-eth/llama2-webui/blob/main/colab/Llama_2_7b_Chat_GPTQ.ipynb)12- Use  [llama2-wrapper](https://pypi.org/project/llama2-wrapper/)  as your local llama2 backend for Generative Agents/Apps; [colab example](./colab/Llama_2_7b_Chat_GPTQ.ipynb).  13- [Run OpenAI Compatible API](https://github.com/liltom-eth/llama2-webui#start-openai-compatible-api) on Llama2 models.14- [News](https://github.com/liltom-eth/llama2-webui/blob/main/docs/news.md), [Benchmark](https://github.com/liltom-eth/llama2-webui/blob/main/docs/performance.md), [Issue Solutions](https://github.com/liltom-eth/llama2-webui/blob/main/docs/issues.md)15 16[llama2-wrapper](https://pypi.org/project/llama2-wrapper/)  is the backend and part of [llama2-webui](https://github.com/liltom-eth/llama2-webui), which can run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac).17 18## Install19 20```bash21pip install llama2-wrapper22```23 24## Start OpenAI Compatible  API25 26```27python -m llama2_wrapper.server28```29 30it will use `llama.cpp` as the backend by default to run `llama-2-7b-chat.ggmlv3.q4_0.bin` model.31 32Start Fast API for `gptq` backend:33 34```35python -m llama2_wrapper.server --backend_type gptq36```37 38Navigate to http://localhost:8000/docs to see the OpenAPI documentation.39 40## API Usage41 42###  `__call__`43 44`__call__()` is the function to generate text from a prompt. 45 46For example, run ggml llama2 model on CPU, [colab example](https://github.com/liltom-eth/llama2-webui/blob/main/colab/ggmlv3_q4_0.ipynb):47 48```python49from llama2_wrapper import LLAMA2_WRAPPER, get_prompt 50llama2_wrapper = LLAMA2_WRAPPER()51# Default running on backend llama.cpp.52# Automatically downloading model to: ./models/llama-2-7b-chat.ggmlv3.q4_0.bin53prompt = "Do you know Pytorch"54# llama2_wrapper() will run __call__()55answer = llama2_wrapper(get_prompt(prompt), temperature=0.9)56```57 58Run gptq llama2 model on Nvidia GPU, [colab example](https://github.com/liltom-eth/llama2-webui/blob/main/colab/Llama_2_7b_Chat_GPTQ.ipynb):59 60```python61from llama2_wrapper import LLAMA2_WRAPPER 62llama2_wrapper = LLAMA2_WRAPPER(backend_type="gptq")63# Automatically downloading model to: ./models/Llama-2-7b-Chat-GPTQ64```65 66Run llama2 7b with bitsandbytes 8 bit with a `model_path`:67 68```python69from llama2_wrapper import LLAMA2_WRAPPER 70llama2_wrapper = LLAMA2_WRAPPER(71	model_path = "./models/Llama-2-7b-chat-hf",72  backend_type = "transformers",73  load_in_8bit = True74)75```76 77### completion78 79  `completion()`  is the function to generate text from a prompt for OpenAI compatible API `/v1/completions`.80 81```python82llama2_wrapper = LLAMA2_WRAPPER()83prompt = get_prompt("Hi do you know Pytorch?")84print(llm.completion(prompt))85```86 87### chat_completion88 89  `chat_completion()`  is the function to generate text from a dialog (chat history) for OpenAI compatible API `/v1/chat/completions`.90 91```python92llama2_wrapper = LLAMA2_WRAPPER()93dialog = [94    {95        "role":"system",96        "content":"You are a helpful, respectful and honest assistant. "97    },{98        "role":"user",99        "content":"Hi do you know Pytorch?",100    },101]102print(llm.chat_completion(dialog))103```104 105### generate106 107`generate()` is the function to create a generator of response from a prompt.108 109This is useful when you want to stream the output like typing in the chatbot.110 111```python112llama2_wrapper = LLAMA2_WRAPPER()113prompt = get_prompt("Hi do you know Pytorch?")114for response in llama2_wrapper.generate(prompt):115	print(response)116 117```118 119The response will be like:120 121```122Yes, 123Yes, I'm 124Yes, I'm familiar 125Yes, I'm familiar with 126Yes, I'm familiar with PyTorch! 127...128```129 130### run131 132`run()` is similar to `generate()`, but `run()`can also accept `chat_history`and `system_prompt` from the users.133 134It will process the input message to llama2 prompt template with `chat_history` and `system_prompt` for a chatbot-like app.135 136### get_prompt137 138`get_prompt()` will process the input message to llama2 prompt with `chat_history` and `system_prompt`for chatbot.139 140By default, `chat_history` and `system_prompt` are empty and `get_prompt()` will add llama2 prompt template to your message:141 142```python143prompt = get_prompt("Hi do you know Pytorch?")144```145 146prompt will be:147 148```149[INST] <<SYS>>150 151<</SYS>>152 153Hi do you know Pytorch? [/INST]154```155 156If use `get_prompt("Hi do you know Pytorch?", system_prompt="You are a helpful...")`:157 158```159[INST] <<SYS>>160You are a helpful, respectful and honest assistant. 161<</SYS>>162 163Hi do you know Pytorch? [/INST]164```165 166### get_prompt_for_dialog167 168`get_prompt_for_dialog()` will process dialog (chat history) to llama2 prompt for OpenAI compatible API `/v1/chat/completions`.169 170```python171dialog = [172    {173        "role":"system",174        "content":"You are a helpful, respectful and honest assistant. "175    },{176        "role":"user",177        "content":"Hi do you know Pytorch?",178    },179]180prompt = get_prompt_for_dialog("Hi do you know Pytorch?")181# [INST] <<SYS>>182# You are a helpful, respectful and honest assistant. 183# <</SYS>>184# 185# Hi do you know Pytorch? [/INST]186```187 188