forestcalled/text-generation-webui
0
1## OpenAI compatible API2 3The main API for this project is meant to be a drop-in replacement to the OpenAI API, including Chat and Completions endpoints. 4 5* It is 100% offline and private.6* It doesn't create any logs.7* It doesn't connect to OpenAI.8* It doesn't use the openai-python library.9 10If you did not use the one-click installers, you may need to install the requirements first:11 12```13pip install -r extensions/openai/requirements.txt14```15 16### Starting the API17 18Add `--api` to your command-line flags.19 20* To create a public Cloudflare URL, add the `--public-api` flag.21* To listen on your local network, add the `--listen` flag.22* To change the port, which is 5000 by default, use `--api-port 1234` (change 1234 to your desired port number).23* To use SSL, add `--ssl-keyfile key.pem --ssl-certfile cert.pem`. Note that it doesn't work with `--public-api`.24* To use an API key for authentication, add `--api-key yourkey`.25 26### Examples27 28For the documentation with all the parameters and their types, consult `http://127.0.0.1:5000/docs` or the [typing.py](https://github.com/oobabooga/text-generation-webui/blob/main/extensions/openai/typing.py) file.29 30The official examples in the [OpenAI documentation](https://platform.openai.com/docs/api-reference) should also work, and the same parameters apply (although the API here has more optional parameters).31 32#### Completions33 34```shell35curl http://127.0.0.1:5000/v1/completions \36 -H "Content-Type: application/json" \37 -d '{38 "prompt": "This is a cake recipe:\n\n1.",39 "max_tokens": 200,40 "temperature": 1,41 "top_p": 0.9,42 "seed": 1043 }'44```45 46#### Chat completions47 48Works best with instruction-following models. If the "instruction_template" variable is not provided, it will be guessed automatically based on the model name using the regex patterns in `models/config.yaml`.49 50```shell51curl http://127.0.0.1:5000/v1/chat/completions \52 -H "Content-Type: application/json" \53 -d '{54 "messages": [55 {56 "role": "user",57 "content": "Hello!"58 }59 ],60 "mode": "instruct",61 "instruction_template": "Alpaca"62 }'63```64 65#### Chat completions with characters66 67```shell68curl http://127.0.0.1:5000/v1/chat/completions \69 -H "Content-Type: application/json" \70 -d '{71 "messages": [72 {73 "role": "user",74 "content": "Hello! Who are you?"75 }76 ],77 "mode": "chat",78 "character": "Example"79 }'80```81 82#### SSE streaming83 84```shell85curl http://127.0.0.1:5000/v1/chat/completions \86 -H "Content-Type: application/json" \87 -d '{88 "messages": [89 {90 "role": "user",91 "content": "Hello!"92 }93 ],94 "mode": "instruct",95 "instruction_template": "Alpaca",96 "stream": true97 }'98```99 100#### Logits101 102```103curl -k http://127.0.0.1:5000/v1/internal/logits \104 -H "Content-Type: application/json" \105 -d '{106 "prompt": "Who is best, Asuka or Rei? Answer:",107 "use_samplers": false108 }'109```110 111#### Logits after sampling parameters112 113```114curl -k http://127.0.0.1:5000/v1/internal/logits \115 -H "Content-Type: application/json" \116 -d '{117 "prompt": "Who is best, Asuka or Rei? Answer:",118 "use_samplers": true,119 "top_k": 3120 }'121```122 123#### Python chat example124 125```python126import requests127 128url = "http://127.0.0.1:5000/v1/chat/completions"129 130headers = {131 "Content-Type": "application/json"132}133 134history = []135 136while True:137 user_message = input("> ")138 history.append({"role": "user", "content": user_message})139 data = {140 "mode": "chat",141 "character": "Example",142 "messages": history143 }144 145 response = requests.post(url, headers=headers, json=data, verify=False)146 assistant_message = response.json()['choices'][0]['message']['content']147 history.append({"role": "assistant", "content": assistant_message})148 print(assistant_message)149```150 151#### Python chat example with streaming152 153Start the script with `python -u` to see the output in real time.154 155```python156import requests157import sseclient # pip install sseclient-py158import json159 160url = "http://127.0.0.1:5000/v1/chat/completions"161 162headers = {163 "Content-Type": "application/json"164}165 166history = []167 168while True:169 user_message = input("> ")170 history.append({"role": "user", "content": user_message})171 data = {172 "mode": "instruct",173 "stream": True,174 "messages": history175 }176 177 stream_response = requests.post(url, headers=headers, json=data, verify=False, stream=True)178 client = sseclient.SSEClient(stream_response)179 180 assistant_message = ''181 for event in client.events():182 payload = json.loads(event.data)183 chunk = payload['choices'][0]['message']['content']184 assistant_message += chunk185 print(chunk, end='')186 187 print()188 history.append({"role": "assistant", "content": assistant_message})189```190 191#### Python completions example with streaming192 193Start the script with `python -u` to see the output in real time.194 195```python196import json197import requests198import sseclient # pip install sseclient-py199 200url = "http://127.0.0.1:5000/v1/completions"201 202headers = {203 "Content-Type": "application/json"204}205 206data = {207 "prompt": "This is a cake recipe:\n\n1.",208 "max_tokens": 200,209 "temperature": 1,210 "top_p": 0.9,211 "seed": 10,212 "stream": True,213}214 215stream_response = requests.post(url, headers=headers, json=data, verify=False, stream=True)216client = sseclient.SSEClient(stream_response)217 218print(data['prompt'], end='')219for event in client.events():220 payload = json.loads(event.data)221 print(payload['choices'][0]['text'], end='')222 223print()224```225 226### Environment variables227 228The following environment variables can be used (they take precendence over everything else):229 230| Variable Name | Description | Example Value |231|------------------------|------------------------------------|----------------------------|232| `OPENEDAI_PORT` | Port number | 5000 |233| `OPENEDAI_CERT_PATH` | SSL certificate file path | cert.pem |234| `OPENEDAI_KEY_PATH` | SSL key file path | key.pem |235| `OPENEDAI_DEBUG` | Enable debugging (set to 1) | 1 |236| `SD_WEBUI_URL` | WebUI URL (used by endpoint) | http://127.0.0.1:7861 |237| `OPENEDAI_EMBEDDING_MODEL` | Embedding model (if applicable) | sentence-transformers/all-mpnet-base-v2 |238| `OPENEDAI_EMBEDDING_DEVICE` | Embedding device (if applicable) | cuda |239 240#### Persistent settings with `settings.yaml`241 242You can also set the following variables in your `settings.yaml` file:243 244```245openai-embedding_device: cuda246openai-embedding_model: "sentence-transformers/all-mpnet-base-v2"247openai-sd_webui_url: http://127.0.0.1:7861248openai-debug: 1249```250 251### Third-party application setup252 253You can usually force an application that uses the OpenAI API to connect to the local API by using the following environment variables:254 255```shell256OPENAI_API_HOST=http://127.0.0.1:5000257```258 259or260 261```shell262OPENAI_API_KEY=sk-111111111111111111111111111111111111111111111111263OPENAI_API_BASE=http://127.0.0.1:5000/v1264```265 266With the [official python openai client](https://github.com/openai/openai-python), the address can be set like this:267 268```python269import openai270 271openai.api_key = "..."272openai.api_base = "http://127.0.0.1:5000/v1"273openai.api_version = "2023-05-15"274```275 276If using .env files to save the `OPENAI_API_BASE` and `OPENAI_API_KEY` variables, make sure the .env file is loaded before the openai module is imported:277 278```python279from dotenv import load_dotenv280load_dotenv() # make sure the environment variables are set before import281import openai282```283 284With the [official Node.js openai client](https://github.com/openai/openai-node) it is slightly more more complex because the environment variables are not used by default, so small source code changes may be required to use the environment variables, like so:285 286```js287const openai = OpenAI(288 Configuration({289 apiKey: process.env.OPENAI_API_KEY,290 basePath: process.env.OPENAI_API_BASE291 })292);293```294 295For apps made with the [chatgpt-api Node.js client library](https://github.com/transitive-bullshit/chatgpt-api):296 297```js298const api = new ChatGPTAPI({299 apiKey: process.env.OPENAI_API_KEY,300 apiBaseUrl: process.env.OPENAI_API_BASE301});302```303### Embeddings (alpha)304 305Embeddings requires `sentence-transformers` installed, but chat and completions will function without it loaded. The embeddings endpoint is currently using the HuggingFace model: `sentence-transformers/all-mpnet-base-v2` for embeddings. This produces 768 dimensional embeddings (the same as the text-davinci-002 embeddings), which is different from OpenAI's current default `text-embedding-ada-002` model which produces 1536 dimensional embeddings. The model is small-ish and fast-ish. This model and embedding size may change in the future.306 307| model name | dimensions | input max tokens | speed | size | Avg. performance |308| ---------------------- | ---------- | ---------------- | ----- | ---- | ---------------- |309| text-embedding-ada-002 | 1536 | 8192 | - | - | - |310| text-davinci-002 | 768 | 2046 | - | - | - |311| all-mpnet-base-v2 | 768 | 384 | 2800 | 420M | 63.3 |312| all-MiniLM-L6-v2 | 384 | 256 | 14200 | 80M | 58.8 |313 314In short, the all-MiniLM-L6-v2 model is 5x faster, 5x smaller ram, 2x smaller storage, and still offers good quality. Stats from (https://www.sbert.net/docs/pretrained_models.html). To change the model from the default you can set the environment variable `OPENEDAI_EMBEDDING_MODEL`, ex. "OPENEDAI_EMBEDDING_MODEL=all-MiniLM-L6-v2".315 316Warning: You cannot mix embeddings from different models even if they have the same dimensions. They are not comparable.317 318### Compatibility & not so compatibility319 320Note: the table below may be obsolete.321 322| API endpoint | tested with | notes |323| ------------------------- | ---------------------------------- | --------------------------------------------------------------------------- |324| /v1/chat/completions | openai.ChatCompletion.create() | Use it with instruction following models |325| /v1/embeddings | openai.Embedding.create() | Using SentenceTransformer embeddings |326| /v1/images/generations | openai.Image.create() | Bare bones, no model configuration, response_format='b64_json' only. |327| /v1/moderations | openai.Moderation.create() | Basic initial support via embeddings |328| /v1/models | openai.Model.list() | Lists models, Currently loaded model first, plus some compatibility options |329| /v1/models/{id} | openai.Model.get() | returns whatever you ask for |330| /v1/edits | openai.Edit.create() | Removed, use /v1/chat/completions instead |331| /v1/text_completion | openai.Completion.create() | Legacy endpoint, variable quality based on the model |332| /v1/completions | openai api completions.create | Legacy endpoint (v0.25) |333| /v1/engines/\*/embeddings | python-openai v0.25 | Legacy endpoint |334| /v1/engines/\*/generate | openai engines.generate | Legacy endpoint |335| /v1/engines | openai engines.list | Legacy Lists models |336| /v1/engines/{model_name} | openai engines.get -i {model_name} | You can use this legacy endpoint to load models via the api or command line |337| /v1/images/edits | openai.Image.create_edit() | not yet supported |338| /v1/images/variations | openai.Image.create_variation() | not yet supported |339| /v1/audio/\* | openai.Audio.\* | supported |340| /v1/files\* | openai.Files.\* | not yet supported |341| /v1/fine-tunes\* | openai.FineTune.\* | not yet supported |342| /v1/search | openai.search, engines.search | not yet supported |343 344#### Applications345 346Almost everything needs the `OPENAI_API_KEY` and `OPENAI_API_BASE` environment variable set, but there are some exceptions.347 348Note: the table below may be obsolete.349 350| Compatibility | Application/Library | Website | Notes |351| ------------- | ---------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |352| ✅❌ | openai-python (v0.25+) | https://github.com/openai/openai-python | only the endpoints from above are working. OPENAI_API_BASE=http://127.0.0.1:5001/v1 |353| ✅❌ | openai-node | https://github.com/openai/openai-node | only the endpoints from above are working. environment variables don't work by default, but can be configured (see above) |354| ✅❌ | chatgpt-api | https://github.com/transitive-bullshit/chatgpt-api | only the endpoints from above are working. environment variables don't work by default, but can be configured (see above) |355| ✅ | anse | https://github.com/anse-app/anse | API Key & URL configurable in UI, Images also work |356| ✅ | shell_gpt | https://github.com/TheR1D/shell_gpt | OPENAI_API_HOST=http://127.0.0.1:5001 |357| ✅ | gpt-shell | https://github.com/jla/gpt-shell | OPENAI_API_BASE=http://127.0.0.1:5001/v1 |358| ✅ | gpt-discord-bot | https://github.com/openai/gpt-discord-bot | OPENAI_API_BASE=http://127.0.0.1:5001/v1 |359| ✅ | OpenAI for Notepad++ | https://github.com/Krazal/nppopenai | api_url=http://127.0.0.1:5001 in the config file, or environment variables |360| ✅ | vscode-openai | https://marketplace.visualstudio.com/items?itemName=AndrewButson.vscode-openai | OPENAI_API_BASE=http://127.0.0.1:5001/v1 |361| ✅❌ | langchain | https://github.com/hwchase17/langchain | OPENAI_API_BASE=http://127.0.0.1:5001/v1 even with a good 30B-4bit model the result is poor so far. It assumes zero shot python/json coding. Some model tailored prompt formatting improves results greatly. |362| ✅❌ | Auto-GPT | https://github.com/Significant-Gravitas/Auto-GPT | OPENAI_API_BASE=http://127.0.0.1:5001/v1 Same issues as langchain. Also assumes a 4k+ context |363| ✅❌ | babyagi | https://github.com/yoheinakajima/babyagi | OPENAI_API_BASE=http://127.0.0.1:5001/v1 |364| ❌ | guidance | https://github.com/microsoft/guidance | logit_bias and logprobs not yet supported |365