forestcalled/text-generation-webui

1## Audio notification2 3If your computer takes a long time to generate each response for the model that you are using, you can enable an audio notification for when the response is completed. This feature was kindly contributed by HappyWorldGames in [#1277](https://github.com/oobabooga/text-generation-webui/pull/1277).4 5### Installation6 7Simply place a file called "notification.mp3" in the same folder as `server.py`. Here you can find some examples:8 9* https://pixabay.com/sound-effects/search/ding/?duration=0-3010* https://pixabay.com/sound-effects/search/notification/?duration=0-3011 12Source: https://github.com/AUTOMATIC1111/stable-diffusion-webui/pull/112613 14This file will be automatically detected the next time you start the web UI.15 16## GPT-4chan17 18[GPT-4chan](https://huggingface.co/ykilcher/gpt-4chan) has been shut down from Hugging Face, so you need to download it elsewhere. You have two options:19 20* Torrent: [16-bit](https://archive.org/details/gpt4chan_model_float16) / [32-bit](https://archive.org/details/gpt4chan_model)21* Direct download: [16-bit](https://theswissbay.ch/pdf/_notpdf_/gpt4chan_model_float16/) / [32-bit](https://theswissbay.ch/pdf/_notpdf_/gpt4chan_model/)22 23The 32-bit version is only relevant if you intend to run the model in CPU mode. Otherwise, you should use the 16-bit version.24 25After downloading the model, follow these steps:26 271. Place the files under `models/gpt4chan_model_float16` or `models/gpt4chan_model`.282. Place GPT-J 6B's config.json file in that same folder: [config.json](https://huggingface.co/EleutherAI/gpt-j-6B/raw/main/config.json).293. Download GPT-J 6B's tokenizer files (they will be automatically detected when you attempt to load GPT-4chan):30 31```32python download-model.py EleutherAI/gpt-j-6B --text-only33```34 35When you load this model in default or notebook modes, the "HTML" tab will show the generated text in 4chan format:36 37![Image3](https://github.com/oobabooga/screenshots/raw/main/gpt4chan.png)38 39## Using LoRAs with GPTQ-for-LLaMa40 41This requires using a monkey patch that is supported by this web UI: https://github.com/johnsmith0031/alpaca_lora_4bit42 43To use it:44 45Install alpaca_lora_4bit using pip46 47```48git clone https://github.com/johnsmith0031/alpaca_lora_4bit.git49cd alpaca_lora_4bit50git fetch origin winglian-setup_pip51git checkout winglian-setup_pip52pip install .53```54 55Start the UI with the --monkey-patch flag:56 57```58python server.py --model llama-7b-4bit-128g --listen --lora tloen_alpaca-lora-7b --monkey-patch59```60 61## DeepSpeed62 63`DeepSpeed ZeRO-3` is an alternative offloading strategy for full-precision (16-bit) transformers models.64 65With this, I have been able to load a 6b model (GPT-J 6B) with less than 6GB of VRAM. The speed of text generation is very decent and much better than what would be accomplished with `--auto-devices --gpu-memory 6`.66 67As far as I know, DeepSpeed is only available for Linux at the moment.68 69### How to use it70 711. Install DeepSpeed: 72 73```74conda install -c conda-forge mpi4py mpich75pip install -U deepspeed76```77 782. Start the web UI replacing `python` with `deepspeed --num_gpus=1` and adding the `--deepspeed` flag. Example:79 80```81deepspeed --num_gpus=1 server.py --deepspeed --chat --model gpt-j-6B82```83 84> RWKV: RNN with Transformer-level LLM Performance85>86> It combines the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding (using the final hidden state).87 88https://github.com/BlinkDL/RWKV-LM89 90https://github.com/BlinkDL/ChatRWKV91 92## Using RWKV in the web UI93 94### Hugging Face weights95 96Simply download the weights from https://huggingface.co/RWKV and load them as you would for any other model.97 98There is a bug in transformers==4.29.2 that prevents RWKV from being loaded in 8-bit mode. You can install the dev branch to solve this bug: `pip install git+https://github.com/huggingface/transformers`99 100### Original .pth weights101 102The instructions below are from before RWKV was supported in transformers, and they are kept for legacy purposes. The old implementation is possibly faster, but it lacks the full range of samplers that the transformers library offers.103 104#### 0. Install the RWKV library105 106```107pip install rwkv108```109 110`0.7.3` was the last version that I tested. If you experience any issues, try ```pip install rwkv==0.7.3```.111 112#### 1. Download the model113 114It is available in different sizes:115 116* https://huggingface.co/BlinkDL/rwkv-4-pile-3b/117* https://huggingface.co/BlinkDL/rwkv-4-pile-7b/118* https://huggingface.co/BlinkDL/rwkv-4-pile-14b/119 120There are also older releases with smaller sizes like:121 122* https://huggingface.co/BlinkDL/rwkv-4-pile-169m/resolve/main/RWKV-4-Pile-169M-20220807-8023.pth123 124Download the chosen `.pth` and put it directly in the `models` folder. 125 126#### 2. Download the tokenizer127 128[20B_tokenizer.json](https://raw.githubusercontent.com/BlinkDL/ChatRWKV/main/v2/20B_tokenizer.json)129 130Also put it directly in the `models` folder. Make sure to not rename it. It should be called `20B_tokenizer.json`.131 132#### 3. Launch the web UI133 134No additional steps are required. Just launch it as you would with any other model.135 136```137python server.py --listen  --no-stream --model RWKV-4-Pile-169M-20220807-8023.pth138```139 140#### Setting a custom strategy141 142It is possible to have very fine control over the offloading and precision for the model with the `--rwkv-strategy` flag. Possible values include:143 144```145"cpu fp32" # CPU mode146"cuda fp16" # GPU mode with float16 precision147"cuda fp16 *30 -> cpu fp32" # GPU+CPU offloading. The higher the number after *, the higher the GPU allocation.148"cuda fp16i8" # GPU mode with 8-bit precision149```150 151See the README for the PyPl package for more details: https://pypi.org/project/rwkv/152 153#### Compiling the CUDA kernel154 155You can compile the CUDA kernel for the model with `--rwkv-cuda-on`. This should improve the performance a lot but I haven't been able to get it to work yet.156 157## Miscellaneous info158 159### You can train LoRAs in CPU mode160 161Load the web UI with162 163```164python server.py --cpu165```166 167and start training the LoRA from the training tab as usual.168 169### You can check the sha256sum of downloaded models with the download script170 171```172python download-model.py facebook/galactica-125m --check173```174 175### The download script continues interrupted downloads by default176 177It doesn't start over.178 179