jbilcke-hf/ai-comic-factory

1---2title: AI Comic Factory3emoji: 👩‍🎨4colorFrom: red5colorTo: yellow6sdk: docker7pinned: true8app_port: 30009disable_embedding: false10short_description: Create your own AI comic with a single prompt11hf_oauth: true12hf_oauth_expiration_minutes: 4320013hf_oauth_scopes: [inference-api]14---15 16# AI Comic Factory17 18Last release: AI Comic Factory 1.219 20The AI Comic Factory has an official website: [aicomicfactory.app](https://aicomicfactory.app)21 22For more information about my other projects please check [linktr.ee/FLNGR](https://linktr.ee/FLNGR).23 24## Funding25 26If you like the AI Comic Factory, let me know!27I am always creating new spaces and exploring new ideas for demos, meaning I don't have much time to take care of all of them (I wish I could clone myself or ask robots to do it).28 29If you appreciate the AI Comic Factory and would like to leave a tip, that would be very kind 🫶30 31<a href="https://www.buymeacoffee.com/flngr" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>32 33## Running the project at home34 35First, I would like to highlight that everything is open-source (see [here](https://huggingface.co/spaces/jbilcke-hf/ai-comic-factory/tree/main), [here](https://huggingface.co/spaces/jbilcke-hf/VideoChain-API/tree/main), [here](https://huggingface.co/spaces/hysts/SD-XL/tree/main), [here](https://github.com/huggingface/text-generation-inference)).36 37However the project isn't a monolithic Space that can be duplicated and ran immediately:38it requires various components to run for the frontend, backend, LLM, SDXL etc.39 40If you try to duplicate the project, open the `.env` you will see it requires some variables.41 42Provider config:43- `LLM_ENGINE`: can be one of `INFERENCE_API`, `INFERENCE_ENDPOINT`, `OPENAI`, `GROQ`, `ANTHROPIC`44- `RENDERING_ENGINE`: can be one of: "INFERENCE_API", "INFERENCE_ENDPOINT", "REPLICATE", "VIDEOCHAIN", "OPENAI" for now, unless you code your custom solution45 46Auth config:47- `AUTH_HF_API_TOKEN`:  if you decide to use Hugging Face for the LLM engine (inference api model or a custom inference endpoint)48- `AUTH_OPENAI_API_KEY`: to use OpenAI for the LLM engine49- `AUTH_GROQ_API_KEY`: to use Groq for the LLM engine50- `AUTH_ANTHROPIC_API_KEY`: to use Anthropic (Claude) for the LLM engine51- `AUTH_VIDEOCHAIN_API_TOKEN`: secret token to access the VideoChain API server52- `AUTH_REPLICATE_API_TOKEN`: in case you want to use Replicate.com53 54Rendering config:55- `RENDERING_HF_INFERENCE_ENDPOINT_URL`: necessary if you decide to use a custom inference endpoint56- `RENDERING_REPLICATE_API_MODEL_VERSION`: url to the VideoChain API server57- `RENDERING_HF_INFERENCE_ENDPOINT_URL`: optional, default to nothing58- `RENDERING_HF_INFERENCE_API_BASE_MODEL`: optional, defaults to "stabilityai/stable-diffusion-xl-base-1.0"59- `RENDERING_HF_INFERENCE_API_REFINER_MODEL`: optional, defaults to "stabilityai/stable-diffusion-xl-refiner-1.0"60- `RENDERING_REPLICATE_API_MODEL`: optional, defaults to "stabilityai/sdxl"61- `RENDERING_REPLICATE_API_MODEL_VERSION`: optional, in case you want to change the version62 63Language model config (depending on the LLM engine you decide to use):64- `LLM_HF_INFERENCE_ENDPOINT_URL`: "<use your own>"65- `LLM_HF_INFERENCE_API_MODEL`: "HuggingFaceH4/zephyr-7b-beta"66- `LLM_OPENAI_API_BASE_URL`: "https://api.openai.com/v1"67- `LLM_OPENAI_API_MODEL`: "gpt-4-turbo"68- `LLM_GROQ_API_MODEL`: "mixtral-8x7b-32768"69- `LLM_ANTHROPIC_API_MODEL`: "claude-3-opus-20240229"70 71In addition, there are some community sharing variables that you can just ignore.72Those variables are not required to run the AI Comic Factory on your own website or computer73(they are meant to create a connection with the Hugging Face community,74and thus only make sense for official Hugging Face apps):75- `NEXT_PUBLIC_ENABLE_COMMUNITY_SHARING`: you don't need this76- `COMMUNITY_API_URL`: you don't need this77- `COMMUNITY_API_TOKEN`: you don't need this78- `COMMUNITY_API_ID`: you don't need this79 80Please read the `.env` default config file for more informations.81To customise a variable locally, you should create a `.env.local`82(do not commit this file as it will contain your secrets).83 84-> If you intend to run it with local, cloud-hosted and/or proprietary models **you are going to need to code 👨‍💻**.85 86## The LLM API (Large Language Model)87 88Currently the AI Comic Factory uses [zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) through an [Inference Endpoint](https://huggingface.co/docs/inference-endpoints/index).89 90You have multiple options:91 92### Option 1: Use an Inference API model93 94This is a new option added recently, where you can use one of the models from the Hugging Face Hub. By default we suggest to use [zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) as it will provide better results than the 7b model.95 96To activate it, create a `.env.local` configuration file:97 98```bash99LLM_ENGINE="INFERENCE_API"100 101HF_API_TOKEN="Your Hugging Face token"102 103# "HuggingFaceH4/zephyr-7b-beta" is used by default, but you can change this104# note: You should use a model able to generate JSON responses,105# so it is storngly suggested to use at least the 34b model106HF_INFERENCE_API_MODEL="HuggingFaceH4/zephyr-7b-beta"107```108 109### Option 2: Use an Inference Endpoint URL110 111If you would like to run the AI Comic Factory on a private LLM running on the Hugging Face Inference Endpoint service, create a `.env.local` configuration file:112 113```bash114LLM_ENGINE="INFERENCE_ENDPOINT"115 116HF_API_TOKEN="Your Hugging Face token"117 118HF_INFERENCE_ENDPOINT_URL="path to your inference endpoint url"119```120 121To run this kind of LLM locally, you can use [TGI](https://github.com/huggingface/text-generation-inference) (Please read [this post](https://github.com/huggingface/text-generation-inference/issues/726) for more information about the licensing).122 123### Option 3: Use an OpenAI API Key124 125This is a new option added recently, where you can use OpenAI API with an OpenAI API Key.126 127To activate it, create a `.env.local` configuration file:128 129```bash130LLM_ENGINE="OPENAI"131 132# default openai api base url is: https://api.openai.com/v1133LLM_OPENAI_API_BASE_URL="A custom OpenAI API Base URL if you have some special privileges"134 135LLM_OPENAI_API_MODEL="gpt-4-turbo"136 137AUTH_OPENAI_API_KEY="Yourown OpenAI API Key"138```139### Option 4: (new, experimental) use Groq140 141```bash142LLM_ENGINE="GROQ"143 144LLM_GROQ_API_MODEL="mixtral-8x7b-32768"145 146AUTH_GROQ_API_KEY="Your own GROQ API Key"147```148### Option 5: (new, experimental) use Anthropic (Claude)149 150```bash151LLM_ENGINE="ANTHROPIC"152 153LLM_ANTHROPIC_API_MODEL="claude-3-opus-20240229"154 155AUTH_ANTHROPIC_API_KEY="Your own ANTHROPIC API Key"156```157 158### Option 6: Fork and modify the code to use a different LLM system159 160Another option could be to disable the LLM completely and replace it with another LLM protocol and/or provider (eg. Claude, Replicate), or a human-generated story instead (by returning mock or static data).161 162### Notes163 164It is possible that I modify the AI Comic Factory to make it easier in the future (eg. add support for Claude or Replicate)165 166## The Rendering API167 168This API is used to generate the panel images. This is an API I created for my various projects at Hugging Face.169 170I haven't written documentation for it yet, but basically it is "just a wrapper ™" around other existing APIs:171 172- The [hysts/SD-XL](https://huggingface.co/spaces/hysts/SD-XL?duplicate=true) Space by [@hysts](https://huggingface.co/hysts)173- And other APIs for making videos, adding audio etc.. but you won't need them for the AI Comic Factory174 175### Option 1: Deploy VideoChain yourself176 177You will have to [clone](https://huggingface.co/spaces/jbilcke-hf/VideoChain-API?duplicate=true) the [source-code](https://huggingface.co/spaces/jbilcke-hf/VideoChain-API/tree/main)178 179Unfortunately, I haven't had the time to write the documentation for VideoChain yet.180(When I do I will update this document to point to the VideoChain's README)181 182 183### Option 2: Use Replicate184 185To use Replicate, create a `.env.local` configuration file:186 187```bash188RENDERING_ENGINE="REPLICATE"189 190RENDERING_REPLICATE_API_MODEL="stabilityai/sdxl"191 192RENDERING_REPLICATE_API_MODEL_VERSION="da77bc59ee60423279fd632efb4795ab731d9e3ca9705ef3341091fb989b7eaf"193 194AUTH_REPLICATE_API_TOKEN="Your Replicate token"195```196 197### Option 3: Use another SDXL API198 199If you fork the project you will be able to modify the code to use the Stable Diffusion technology of your choice (local, open-source, proprietary, your custom HF Space etc).200 201It would even be something else, such as Dall-E.202