jbilcke-hf/ai-comic-factory

1---2title: AI Comic Factory3emoji: 👩‍🎨4colorFrom: red5colorTo: yellow6sdk: docker7pinned: true8app_port: 30009disable_embedding: false10short_description: Create your own AI comic with a single prompt11hf_oauth: true12hf_oauth_expiration_minutes: 4320013hf_oauth_scopes: [inference-api]14---15 16# AI Comic Factory17 18Last release: AI Comic Factory 1.719 20*SDXL is not supported anymore, please use Flux.1, thank you.*21 22The AI Comic Factory will soon have an official website: [aicomicfactory.app](https://aicomicfactory.app)23 24For more information about my other projects please check [linktr.ee/FLNGR](https://linktr.ee/FLNGR).25 26## Running the project at home27 28The project requires various components to run for the frontend, backend, LLM, SDXL etc.29 30If you try to duplicate the project, open the `.env` you will see it requires some variables.31 32Provider config:33- `LLM_ENGINE`: can be one of `INFERENCE_API`, `INFERENCE_ENDPOINT`, `OPENAI`, `GROQ`, `ANTHROPIC`34- `RENDERING_ENGINE`: can be one of: "INFERENCE_API", "INFERENCE_ENDPOINT", "REPLICATE", "OPENAI" for now, unless you code your custom solution35 36Auth config:37- `AUTH_HF_API_TOKEN`:  if you decide to use Hugging Face for the LLM engine (inference api model or a custom inference endpoint)38- `AUTH_OPENAI_API_KEY`: to use OpenAI for the LLM engine39- `AUTH_GROQ_API_KEY`: to use Groq for the LLM engine40- `AUTH_ANTHROPIC_API_KEY`: to use Anthropic (Claude) for the LLM engine41- `AUTH_REPLICATE_API_TOKEN`: in case you want to use Replicate.com42 43Rendering config:44- `RENDERING_HF_INFERENCE_ENDPOINT_URL`: necessary if you decide to use a custom inference endpoint45- `RENDERING_HF_INFERENCE_ENDPOINT_URL`: optional, default to nothing46- `RENDERING_HF_INFERENCE_API_BASE_MODEL`: optional, defaults to "stabilityai/stable-diffusion-xl-base-1.0"47- `RENDERING_HF_INFERENCE_API_REFINER_MODEL`: optional, defaults to "stabilityai/stable-diffusion-xl-refiner-1.0"48- `RENDERING_REPLICATE_API_MODEL`: optional, defaults to "stabilityai/sdxl"49- `RENDERING_REPLICATE_API_MODEL_VERSION`: optional, in case you want to change the version50 51Language model config (depending on the LLM engine you decide to use):52- `LLM_HF_INFERENCE_ENDPOINT_URL`: "<use your own>"53- `LLM_HF_INFERENCE_API_MODEL`: "HuggingFaceH4/zephyr-7b-beta"54- `LLM_OPENAI_API_BASE_URL`: "https://api.openai.com/v1"55- `LLM_OPENAI_API_MODEL`: "gpt-4-turbo"56- `LLM_GROQ_API_MODEL`: "mixtral-8x7b-32768"57- `LLM_ANTHROPIC_API_MODEL`: "claude-3-opus-20240229"58 59In addition, there are some community sharing variables that you can just ignore.60Those variables are not required to run the AI Comic Factory on your own website or computer61(they are meant to create a connection with the Hugging Face community,62and thus only make sense for official Hugging Face apps):63- `NEXT_PUBLIC_ENABLE_COMMUNITY_SHARING`: you don't need this64- `COMMUNITY_API_URL`: you don't need this65- `COMMUNITY_API_TOKEN`: you don't need this66- `COMMUNITY_API_ID`: you don't need this67 68Please read the `.env` default config file for more informations.69To customise a variable locally, you should create a `.env.local`70(do not commit this file as it will contain your secrets).71 72-> If you intend to run it with local, cloud-hosted and/or proprietary models **you are going to need to code 👨‍💻**.73 74## The LLM API (Large Language Model)75 76Currently the AI Comic Factory uses [zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) through an [Inference Endpoint](https://huggingface.co/docs/inference-endpoints/index).77 78You have multiple options:79 80### Option 1: Use an Inference API model81 82This is a new option added recently, where you can use one of the models from the Hugging Face Hub. By default we suggest to use [zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) as it will provide better results than the 7b model.83 84To activate it, create a `.env.local` configuration file:85 86```bash87LLM_ENGINE="INFERENCE_API"88 89HF_API_TOKEN="Your Hugging Face token"90 91# "HuggingFaceH4/zephyr-7b-beta" is used by default, but you can change this92# note: You should use a model able to generate JSON responses,93# so it is storngly suggested to use at least the 34b model94HF_INFERENCE_API_MODEL="HuggingFaceH4/zephyr-7b-beta"95```96 97### Option 2: Use an Inference Endpoint URL98 99If you would like to run the AI Comic Factory on a private LLM running on the Hugging Face Inference Endpoint service, create a `.env.local` configuration file:100 101```bash102LLM_ENGINE="INFERENCE_ENDPOINT"103 104HF_API_TOKEN="Your Hugging Face token"105 106HF_INFERENCE_ENDPOINT_URL="path to your inference endpoint url"107```108 109To run this kind of LLM locally, you can use [TGI](https://github.com/huggingface/text-generation-inference) (Please read [this post](https://github.com/huggingface/text-generation-inference/issues/726) for more information about the licensing).110 111### Option 3: Use an OpenAI API Key112 113This is a new option added recently, where you can use OpenAI API with an OpenAI API Key.114 115To activate it, create a `.env.local` configuration file:116 117```bash118LLM_ENGINE="OPENAI"119 120# default openai api base url is: https://api.openai.com/v1121LLM_OPENAI_API_BASE_URL="A custom OpenAI API Base URL if you have some special privileges"122 123LLM_OPENAI_API_MODEL="gpt-4-turbo"124 125AUTH_OPENAI_API_KEY="Yourown OpenAI API Key"126```127### Option 4: (new, experimental) use Groq128 129```bash130LLM_ENGINE="GROQ"131 132LLM_GROQ_API_MODEL="mixtral-8x7b-32768"133 134AUTH_GROQ_API_KEY="Your own GROQ API Key"135```136### Option 5: (new, experimental) use Anthropic (Claude)137 138```bash139LLM_ENGINE="ANTHROPIC"140 141LLM_ANTHROPIC_API_MODEL="claude-3-opus-20240229"142 143AUTH_ANTHROPIC_API_KEY="Your own ANTHROPIC API Key"144```145 146### Option 6: Fork and modify the code to use a different LLM system147 148Another option could be to disable the LLM completely and replace it with another LLM protocol and/or provider (eg. Claude, Replicate), or a human-generated story instead (by returning mock or static data).149 150### Notes151 152It is possible that I modify the AI Comic Factory to make it easier in the future (eg. add support for Claude or Replicate)153 154## Rendering Images155 156This API is used to generate the panel images. This is an API I created for my various projects at Hugging Face.157 158I haven't written documentation for it yet, but basically it is "just a wrapper ™" around other existing APIs:159 160- The [hysts/SD-XL](https://huggingface.co/spaces/hysts/SD-XL?duplicate=true) Space by [@hysts](https://huggingface.co/hysts)161- And other APIs for making videos, adding audio etc.. but you won't need them for the AI Comic Factory162 163### Option 1:  Use the Hugging Face Inference API164 165This is the default mode. You only need a valid Hugging Face token to use the API.166 167### Option 2: Use Replicate168 169To use Replicate, create a `.env.local` configuration file:170 171```bash172RENDERING_ENGINE="REPLICATE"173 174RENDERING_REPLICATE_API_MODEL="stabilityai/sdxl"175 176RENDERING_REPLICATE_API_MODEL_VERSION="da77bc59ee60423279fd632efb4795ab731d9e3ca9705ef3341091fb989b7eaf"177 178AUTH_REPLICATE_API_TOKEN="Your Replicate token"179```180 181### Option 3: Use another SDXL API182 183If you fork the project you will be able to modify the code to use the Stable Diffusion technology of your choice (local, open-source, proprietary, your custom HF Space etc).184 185It would even be something else, such as Dall-E.186