jbilcke-hf/ai-comic-factory

1# AI Comic Factory - Project Documentation2 3## Project Overview4 5**AI Comic Factory** is a Next.js application that generates AI-powered comic strips using Large Language Models (LLMs) and image generation APIs. Users input a prompt, select a comic style, and the system generates a complete comic with panels, dialog, and artwork.6 7**Key Features:**8- Generate complete comics from a single text prompt9- Multiple comic art styles and fonts10- Support for multiple LLM providers (OpenAI, Anthropic, Groq, Hugging Face)11- Multiple image generation engines (SDXL, OpenAI DALL-E, Replicate)12- Interactive comic editor with speech bubbles and captions13- Export to CLAP format (Cinematic Language and Audio Protocol)14- Community sharing features (optional)15- OAuth integration with Hugging Face16 17## Technology Stack18 19**Frontend:**20- Next.js 14.2.7 with App Router21- React 18.3.1 with TypeScript 5.4.522- Tailwind CSS 3.4.1 with custom comic fonts23- shadcn/ui component library (Radix UI primitives)24- Zustand for state management25- React Konva for canvas-based comic editing26- Framer Motion alternatives via Tailwind animations27 28**Backend/API:**29- Next.js Server Actions (9 server functions identified)30- Multiple LLM integrations: OpenAI, Anthropic Claude, Groq, Hugging Face31- Multiple rendering engines: SDXL, Replicate, VideoChain API, OpenAI DALL-E32- Image processing with Sharp, HTML2Canvas33- Docker containerization34 35**Key Dependencies:**36- `@aitube/clap` - CLAP format support for multimedia projects37- `@anthropic-ai/sdk` - Claude AI integration38- `@huggingface/inference` - Hugging Face model access39- `groq-sdk` - Groq API integration40- `openai` - OpenAI API integration41- `replicate` - Replicate.com API integration42- Custom font handling with 13 different comic fonts43 44## Project Structure45 46```47src/48├── app/                      # Next.js app router49│   ├── engine/              # Core business logic50│   │   ├── presets.ts       # Comic style presets (678 lines, 4 main presets)51│   │   ├── render.ts        # Image generation engine52│   │   ├── caption.ts       # Caption generation53│   │   └── censorship.ts    # Content filtering54│   ├── interface/           # UI components (22 directories)55│   │   ├── page/           # Comic page layout56│   │   ├── panel/          # Individual comic panels57│   │   ├── bottom-bar/     # Controls and actions58│   │   ├── settings-dialog/ # Configuration UI59│   │   └── ...60│   ├── queries/            # Server-side data fetching (13 files)61│   │   ├── predict.ts      # LLM prediction orchestration62│   │   ├── predictNextPanels.ts # Panel generation logic63│   │   ├── predictWith*.ts # Provider-specific implementations64│   │   └── ...65│   ├── store/              # Zustand state management66│   │   └── index.ts        # Main app state (21KB)67│   ├── layouts/            # Comic layout definitions68│   └── main.tsx           # Main application component69├── components/70│   ├── ui/                 # shadcn/ui components (27 components)71│   └── icons/             # Custom icons72├── lib/                    # Utility functions (49 files)73│   ├── fonts.ts           # Comic font definitions74│   ├── bubble/            # Speech bubble utilities75│   └── [various utilities for image processing, parsing, etc.]76├── fonts/                  # 13 custom comic fonts77└── types.ts               # TypeScript type definitions (217 lines)78```79 80## Development Commands81 82```bash83# Development84npm run dev          # Start development server85npm run build        # Production build86npm run start        # Start production server87npm run lint         # ESLint checking88 89# Node version90nvm use             # Uses Node v20.17.0 (specified in .nvmrc)91```92 93## Environment Configuration94 95The application requires extensive environment configuration in `.env.local`:96 97**Core Engines:**98- `LLM_ENGINE`: "INFERENCE_API" | "INFERENCE_ENDPOINT" | "OPENAI" | "GROQ" | "ANTHROPIC"99- `RENDERING_ENGINE`: "INFERENCE_API" | "INFERENCE_ENDPOINT" | "REPLICATE" | "VIDEOCHAIN" | "OPENAI"100 101**Authentication (configure only what you use):**102- `AUTH_HF_API_TOKEN` - Hugging Face API token103- `AUTH_OPENAI_API_KEY` - OpenAI API key104- `AUTH_GROQ_API_KEY` - Groq API key105- `AUTH_ANTHROPIC_API_KEY` - Anthropic/Claude API key106- `AUTH_REPLICATE_API_TOKEN` - Replicate.com token107- `AUTH_VIDEOCHAIN_API_TOKEN` - VideoChain API token108 109**LLM Configuration:**110- `LLM_HF_INFERENCE_API_MODEL` - Default: "HuggingFaceH4/zephyr-7b-beta"111- `LLM_OPENAI_API_MODEL` - Default: "gpt-4-turbo"112- `LLM_GROQ_API_MODEL` - Default: "mixtral-8x7b-32768"113- `LLM_ANTHROPIC_API_MODEL` - Default: "claude-3-opus-20240229"114 115**Rendering Configuration:**116- `RENDERING_HF_INFERENCE_API_BASE_MODEL` - Default: "stabilityai/stable-diffusion-xl-base-1.0"117- `RENDERING_REPLICATE_API_MODEL` - Default: "stabilityai/sdxl"118- `MAX_NB_PAGES` - Default: 6119 120## Architecture Patterns121 122**State Management:**123- Zustand store with typed selectors and actions124- Complex state includes: panels, speeches, captions, renderedScenes, layouts125- Real-time panel generation status tracking126 127**LLM Integration Pattern:**128- Abstracted provider interface through `predict()` function129- Provider-specific implementations in separate files130- Standardized prompt templates and response parsing131- Support for multiple prompt formats (Zephyr, Llama, etc.)132 133**Image Generation Flow:**1341. User provides prompt + selects preset1352. LLM generates panel descriptions, speech, and captions1363. Each panel description is sent to rendering engine1374. Images are generated and cached1385. User can edit speech bubbles and captions1396. Final comic can be exported as image or CLAP file140 141**Server Actions Architecture:**142- 9 server actions for LLM predictions and rendering143- Clean separation between UI and server logic144- Error handling and fallbacks for API failures145 146**Comic Preset System:**147- 4 main preset categories with 678 lines of configuration148- Each preset defines: art style, color scheme, font, LLM prompts, image prompts149- Extensible system for adding new comic styles150 151**Font System:**152- 13 custom comic fonts loaded as local fonts153- Includes both Google Fonts (Indie Flower, The Girl Next Door) and custom fonts154- Proper CSS variable integration for consistent typography155 156## Key Business Logic157 158**Panel Generation (`predictNextPanels`):**159- Generates multiple comic panels from a single prompt160- Handles continuation of existing stories161- Parses LLM responses into structured panel data (instructions, speech, captions)162- Error recovery and retry logic163 164**Rendering Pipeline (`render.ts`):**165- Multi-provider image generation (Replicate, HF, OpenAI, VideoChain)166- Automatic fallbacks between providers167- Image caching and optimization168- Support for different aspect ratios and resolutions169 170**State Persistence:**171- LocalStorage integration for user settings172- CLAP file format support for project serialization173- OAuth state management with Hugging Face174 175## Development Patterns176 177**Component Organization:**178- Feature-based component structure in `app/interface/`179- Reusable UI components in `components/ui/`180- Custom hooks in `lib/` for complex logic181 182**Type Safety:**183- Comprehensive TypeScript definitions in `types.ts`184- Strict typing for LLM engines, rendering engines, and data flows185- Generic interfaces for extensible provider support186 187**Error Handling:**188- Graceful degradation for API failures189- User feedback through toast notifications190- Fallback content for missing images/data191 192**Performance Considerations:**193- Image optimization with Sharp194- Lazy loading of comic panels195- Efficient state updates with Zustand196- Canvas-based rendering for complex layouts197 198## Testing & Quality199 200- **Linting**: ESLint with Next.js configuration201- **No test files found** - this is an area for improvement202- **Type checking**: Strict TypeScript configuration203- **Docker**: Production containerization available204 205## Deployment206 207- Designed for Hugging Face Spaces deployment208- Docker containerization with Node.js Alpine209- Standalone Next.js output for containerized deployment210- Environment-based configuration for different deployment targets211 212## Community & Contributions213 214- Open source project on Hugging Face215- Community contributions documented in `CONTRIBUTORS.md`216- Optional community sharing features217- OAuth integration for user management218 219## Development Notes220 221- **No API routes found** - uses Server Actions exclusively222- **Canvas-based editing** with React Konva for interactive panels223- **Multi-provider architecture** allows switching between AI services224- **Extensive font library** for authentic comic typography225- **CLAP format integration** for multimedia project export226- **Rate limiting** configurable for production usage227 228## Quick Start for Developers229 2301. Copy `.env` to `.env.local` and configure your API keys2312. Choose your LLM_ENGINE and RENDERING_ENGINE2323. Install dependencies: `npm install`2334. Run development server: `npm run dev`2345. The app will guide you through first-time setup235 236Most common development setup:237- LLM_ENGINE: "OPENAI" with OpenAI API key238- RENDERING_ENGINE: "REPLICATE" with Replicate token239- This provides reliable, high-quality results for testing