aaditech/demo-chatgpt

Visual ChatGPT

Visual ChatGPT connects ChatGPT and a series of Visual Foundation Models to enable sending and receiving images during chatting.

See our paper: <font size=5>Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models</font>

Visual ChatGPT Colab Support

You can run the colab notebook with following models text to image,ImageCaptioning,BLIP VQA,image to canny 

Demo

<img src="./assets/demo_short.gif" width="750">

System Architecture

<p align="center"><img src="./assets/figure.jpg" alt="Logo"></p>

Quick Start

# create a new environment
conda create -n visgpt python=3.8

# activate the new environment
conda activate visgpt

#  prepare the basic environments
pip install -r requirement.txt

# download the visual foundation models
bash download.sh

# prepare your private openAI private key
export OPENAI_API_KEY={Your_Private_Openai_Key}

# create a folder to save images
mkdir ./image

# Start Visual ChatGPT !
python visual_chatgpt.py

GPU memory usage

Here we list the GPU memory usage of each visual foundation model, one can modify `self.tools` with fewer visual foundation models to save your GPU memory:

Acknowledgement

We appreciate the open source of the following projects:

• —HuggingFace [[Project]](https://github.com/huggingface/transformers)

• —ControlNet [[Paper]](https://arxiv.org/abs/2302.05543) [[Project]](https://github.com/lllyasviel/ControlNet)

• —Stable Diffusion [[Paper]](https://arxiv.org/abs/2112.10752) [[Project]](https://github.com/CompVis/stable-diffusion)