forestcalled/text-generation-webui
0
1Used to have multi-turn conversations with the model.2 3## Input area4 5The following buttons can be found. Note that the hover menu can be replaced with always-visible buttons with the `--chat-buttons` flag.6 7* **Generate**: sends your message and makes the model start a reply.8* **Stop**: stops an ongoing generation as soon as the next token is generated (which can take a while for a slow model).9* **Continue**: makes the model attempt to continue the existing reply. In some cases, the model may simply end the existing turn immediately without generating anything new, but in other cases, it may generate a longer reply.10* **Regenerate**: similar to Generate, but your last message is used as input instead of the text in the input field. Note that if the temperature/top_p/top_k parameters are low in the "Parameters" tab of the UI, the new reply may end up identical to the previous one.11* **Remove last reply**: removes the last input/output pair from the history and sends your last message back into the input field.12* **Replace last reply**: replaces the last reply with whatever you typed into the input field. Useful in conjunction with "Copy last reply" if you want to edit the bot response.13* **Copy last reply**: sends the contents of the bot's last reply to the input field.14* **Impersonate**: makes the model generate a new message on your behalf in the input field, taking into consideration the existing chat history.15* **Send dummy message**: adds a new message to the chat history without causing the model to generate a reply.16* **Send dummy reply**: adds a new reply to the chat history as if the model had generated this reply. Useful in conjunction with "Send dummy message".17* **Start new chat**: starts a new conversation while keeping the old one saved. If you are talking to a character that has a "Greeting" message defined, this message will be automatically added to the new history.18* **Send to default**: sends the entire chat prompt up to now to the "Default" tab.19* **Send to notebook**: sends the entire chat prompt up to now to the "Notebook" tab.20 21The **Show controls** checkbox causes the input fields below the input textbox to disappear. It is useful for making the page fit entirely into view and not scroll.22 23## Past chats24 25Allows you to switch between the current and previous conversations with the current character, or between the current and previous instruct conversations (if in "instruct" mode). The **Rename** menu can be used to give a unique name to the selected conversation, and the 🗑️ button allows you to delete it.26 27## Start reply with28 29Whatever you type there will appear at the start of every reply by the bot. This is useful to guide the response in the desired direction.30 31## Mode32 33The most important input field. It defines how the chat prompt is formatted. There are 3 options: chat, chat-instruct, and instruct. It is worth going into more detail about this because it seems to not be obvious to a lot of people.34 35### Instruction-following models36 37There are two kinds of models: base models, like Llama and GPT-J, and fine-tuned models, like Alpaca and Vicuna. Fine-tuned models are trained starting from base models, most often with the goal of getting the model to understand and respond to instructions just like ChatGPT does. Let's call such models *instruction-following models*.38 39Each instruction-following model was trained on a specific prompt format, and you have to use that exact prompt format if you want the model to follow your instructions as accurately as it can.40 41As an example, this is the Alpaca format:42 43```44Below is an instruction that describes a task. Write a response that appropriately completes the request.45 46### Instruction:47Hi there!48 49### Response:50Hello! It's nice to meet you. What can I help with?51 52### Instruction:53How are you?54 55### Response:56I'm doing well, thank you for asking! Is there something specific you would like to talk about or ask me? I'm here to help answer any questions you may have.57```58 59This format is characterized by a context string at the top, and alternating turns where each user input starts with `### Instruction:` and each bot turn starts with `### Response:`. There are also weirder formats, like the one used by the Llama-2-chat models released by Meta AI:60 61```62[INST] <<SYS>>63Answer the questions.64<</SYS>>65Hi there! [/INST] Hello! It's nice to meet you. What can I help with? </s><s>[INST] How are you? [/INST] I'm doing well, thank you for asking! Is there something specific you would like to talk about or ask me? I'm here to help answer any questions you may have.66```67 68In this format, there are special tokens at the end of each bot reply (`</s>`, the end of sequence token, and `<s>`, the beginning of sequence token); no new lines separating the turns; and the context string is written between `<<SYS>>` and `<</SYS>>`. Despite the intimidating look of this format, the logic is the same: there are user turns and bot turns, and each one appears in a specific place in the template.69 70It is important to emphasize that instruction-following models **have to be used with the exact prompt format that they were trained on**. Using those models with any other prompt format should be considered undefined behavior. The model will still generate replies, but they will be less accurate to your inputs.71 72Now that an instruction-following model is defined, we can move on to describing the 3 chat modes.73 74### Chat75 76Used for talking to the character defined under "Parameters" > "Character" using a simple chat prompt in this format:77 78```79Chiharu Yamada's Persona: Chiharu Yamada is a young, computer engineer-nerd with a knack for problem solving and a passion for technology.80You: Hi there!81Chiharu Yamada: Hello! It's nice to meet you. What can I help with?82You: How are you?83Chiharu Yamada: I'm doing well, thank you for asking! Is there something specific you would like to talk about or ask me? I'm here to help answer any questions you may have.84```85 86There are 3 adjustable parameters in "Parameters" > "Character" being used in this prompt:87 88* The **Context** string appears at the top of the prompt. Most often it describes the bot's personality and adds a few example messages to guide the model towards the desired reply length and format. This string never gets truncated: as the prompt size increases, old messages get removed one at a time until the prompt becomes smaller than the truncation length set under "Parameters" > "Generation" > "Truncate the prompt up to this length".89* The **Your name** string appears at the beginning of each user reply. By default, this string is "You".90* The **Character's name** string appears at the beginning of each bot reply.91 92Additionally, the **Greeting** string appears as the bot's opening message whenever the history is cleared.93 94The "Chat" option should typically be used only for base models or non-instruct fine tunes, and should not be used for instruction-following models.95 96### Instruct97 98Used for talking to an instruction-following model using the prompt format defined under "Parameters" > "Instruction template". Think of this option as an offline ChatGPT.99 100The prompt format is defined by the **Instruction template** parameter in "Parameters" > "Instruction template", which represents a Jinja2 template.101 102Note that when you load a model in the "Model" tab, the web UI will try to automatically detect its instruction template (if any), and will update the values under "Parameters" > "Instruction template" accordingly. This is done using a set of regular expressions defined in `models/config.yaml`. This detection is not guaranteed to be accurate. You should check the model card on Hugging Face to see if you are using the correct prompt format.103 104### Chat-instruct105 106As said above, instruction-following models are meant to be used with their specific prompt templates. The chat-instruct mode allows you to use those templates to generate a chat reply, thus mixing Chat and Instruct modes (hence the name).107 108It works by creating a single instruction-following turn where a command is given followed by the regular chat prompt. Here is an example in Alpaca format:109 110```111Below is an instruction that describes a task. Write a response that appropriately completes the request.112 113### Instruction:114Continue the chat dialogue below. Write a single reply for the character "Chiharu Yamada".115Chiharu Yamada's Persona: Chiharu Yamada is a young, computer engineer-nerd with a knack for problem solving and a passion for technology.116You: Hi there!117Chiharu Yamada: Hello! It's nice to meet you. What can I help with?118You: How are you?119 120### Response:121Chiharu Yamada:122```123 124Here, the command is125 126> Continue the chat dialogue below. Write a single reply for the character "Chiharu Yamada".127 128Below this command, the regular chat prompt is added, including its Context string and the chat history, and then the user turn ends. The bot turn starts with the "Character's name" string followed by `:`, thus prompting the instruction-following model to write a single reply for the character.129 130The chat-instruct command can be customized under "Parameters" > "Instruction template" > "Command for chat-instruct mode". Inside that command string, `<|character|>` is a placeholder that gets replaced with the bot name, and `<|prompt|>` is a placeholder that gets replaced with the full chat prompt.131 132Note that you can get creative: instead of writing something trivial like "Write a single reply for the character", you could add more complex instructions like133 134> This is an adventure game, and your task is to write a reply in name of "<|character|>" where 3 options are given for the user to then choose from.135 136And it works:137 138139 140## Chat style141 142This defines the visual style of the chat UI. Each option is a CSS file defined under `text-generation-webui/css/chat_style-name.css`, where "name" is how this style is called in the dropdown menu. You can add new styles by simply copying `chat_style-cai-chat.css` to `chat_style-myNewStyle.css` and editing the contents of this new file. If you end up with a style that you like, you are highly encouraged to submit it to the repository.143 144The styles are only applied to chat and chat-instruct modes. Instruct mode has its separate style defined in `text-generation-webui/css/html_instruct_style.css`.145 146## Character gallery147 148This menu is a built-in extension defined under `text-generation-webui/extensions/gallery`. It displays a gallery with your characters, and if you click on a character, it will be automatically selected in the menu under "Parameters" > "Character".149