KillerKing93/Transformers-InferenceServer-OpenAPI

1# Architecture (Python FastAPI + Transformers)2 3This document describes the Python-based, OpenAI-compatible inference server for Qwen3-4B-Instruct, designed to power an **AI-powered marketplace intelligence system**.4 5## System Purpose6 7This inference server serves as the AI backend for a smart marketplace platform where:8- Suppliers can register and list products9- Users query product availability and get AI recommendations10- Products are matched based on user location to find nearest suppliers11- AI assistant helps with natural language product discovery12 13Key source files14- Server entry: [main.py](main.py)15- Inference engine: [Python.class Engine](main.py:464)16- Endpoints: Health [Python.app.get()](main.py:577), Chat Completions [Python.app.post()](main.py:591), Cancel [Python.app.post()](main.py:792)17- Streaming + resume: [Python.class _SSESession](main.py:435), [Python.class _SessionStore](main.py:449), [Python.class _SQLiteStore](main.py:482), [Python.function chat_completions](main.py:591)18- Local run (uvicorn): [Python.main()](main.py:807)19- Configuration template: [.env.example](.env.example)20- Dependencies: [requirements.txt](requirements.txt)21 22Model target (default)23- Hugging Face: unsloth/Qwen3-4B-Instruct-2507 (Transformers, text-only instruct model)24- Overridable via environment variable: MODEL_REPO_ID25 26**Deprecated features**: Multimodal parsing (images/videos) and KTP OCR endpoint are deprecated as of migration to text-only model. Code remains for reference but is non-functional.27 28## Overview29 30The server exposes an OpenAI-compatible endpoint for chat completions:31- **Text-only prompts** (primary use case for marketplace AI assistant)32- Non-streaming JSON responses33- Streaming via Server-Sent Events (SSE) with resumable delivery using Last-Event-ID34- Resumability is achieved with an in‑memory ring buffer and optional SQLite persistence35 36## Components37 381) FastAPI application39- Instantiated in [Python.main module](main.py:541) and endpoints mounted at:40  - Health: [Python.app.get()](main.py:577)41  - Chat Completions (non-stream + SSE): [Python.app.post()](main.py:591) - **Primary endpoint for marketplace AI**42  - Manual cancel (custom): [Python.app.post()](main.py:792)43  - ~~KTP OCR: [Python.app.post()](main.py:1310)~~ - **DEPRECATED** (requires multimodal model)44- CORS is enabled for simplicity.45 462) Inference Engine (Transformers)47- Class: [Python.class Engine](main.py:464)48- Loads:49  - Processor: AutoProcessor(trust_remote_code=True)50  - Model: AutoModelForCausalLM (device_map, dtype configurable via env)51- Core methods:52  - Text-only generate: [Python.function infer](main.py:326)53  - Streaming generate (iterator): [Python.function infer_stream](main.py:375)54 553) ~~Multimodal preprocessing~~ - **DEPRECATED**56- ~~Images/Videos processing~~ - Code remains but is non-functional with text-only model57- For marketplace use case, all interactions are text-based:58  - Product queries: "laptop gaming Jakarta"59  - Recommendations: "laptop programming 10 juta"60  - Location queries: supplier location data passed as text in conversation context61 624) SSE streaming with resume63- Session objects:64  - [Python.class _SSESession](main.py:435): ring buffer, condition variable, producer thread reference, cancellation event, listener count, and disconnect timer65  - [Python.class _SessionStore](main.py:449): in-memory map with TTL + GC66  - Optional persistence: [Python.class _SQLiteStore](main.py:482) for replaying chunks across restarts67- SSE id format: "session_id:index"68- Resume:69  - Client sends Last-Event-ID header (or query ?last_event_id=...) and the same session_id in the body70  - Server replays cached/persisted chunks after the provided index, then continues live streaming71- Producer:72  - Created on demand per session; runs generation in a daemon thread and pushes chunks into the ring buffer and SQLite (if enabled)73  - See producer closure inside [Python.function chat_completions](main.py:591)74- Auto-cancel on disconnect:75  - If all clients disconnect for CANCEL_AFTER_DISCONNECT_SECONDS (default 3600s), a timer signals cancellation via a stopping criteria in [Python.function infer_stream](main.py:375)76 77## Request flow78 79Non-streaming (POST /v1/chat/completions)801. Validate input, load engine singleton via [Python.function get_engine](main.py:558)812. Convert OpenAI-style messages to Qwen chat template via apply_chat_template823. ~~Preprocess images/videos~~ - DEPRECATED (text-only model)834. Generate with [Python.function infer](main.py:326)845. Return OpenAI-compatible response (choices[0].message.content)85 86Streaming (POST /v1/chat/completions with "stream": true)871. Determine session_id:88   - Use body.session_id if provided; otherwise generated server-side892. Parse Last-Event-ID (or query ?last_event_id) to get last delivered index903. Create/start or reuse producer thread for this session914. StreamingResponse generator:92   - Replays persisted events (SQLite, if enabled) and in-memory buffer after last index93   - Waits on condition variable for new tokens94   - Emits "[DONE]" at the end or upon buffer completion955. Clients can reconnect and resume by sending Last-Event-ID: "session_id:index"966. If all clients disconnect, an auto-cancel timer can stop generation (configurable via env)97 98Manual cancel (POST /v1/cancel/{session_id})99- Custom operational shortcut to cancel an in-flight generation for a session id.100- This is not part of the legacy OpenAI Chat Completions spec (OpenAI’s newer Responses API defines cancel); it is provided for practical control.101 102KTP OCR (POST /ktp-ocr/)103- Specialized endpoint for Indonesian ID card (KTP) optical character recognition.104- Accepts multipart form-data with image file, extracts structured JSON data using multimodal inference.105- Returns standardized fields: nik, nama, tempat_lahir, tgl_lahir, jenis_kelamin, alamat (with nested fields), agama, status_perkawinan, pekerjaan, kewarganegaraan, berlaku_hingga.106- Uses custom prompt engineering for accurate structured extraction from Qwen3-VL model.107- Inspired by raflyryhnsyh/Gemini-OCR-KTP but adapted for local, self-hosted inference.108 109## Message and content mapping110 111Input format (OpenAI-like):112- "messages" list of role/content entries113- content can be:114  - string (text)115  - array of parts with "type":116    - "text": { text: "..."}117    - "image_url": { image_url: { url: "..." } } or { image_url: "..." }118    - "input_image": { b64_json: "..." } or { image: "..." }119    - "video_url": { video_url: { url: "..." } } or { video_url: "..." }120    - "input_video": { b64_json: "..." } or { video: "..." }121 122Conversion:123- [Python.function build_mm_messages](main.py:251) constructs a multimodal content list per message:124  - { type: "text", text: ... }125  - { type: "image", image: PIL.Image }126  - { type: "video", video: [PIL.Image frames] }127 128Template:129- Qwen apply_chat_template:130  - See usage in [Python.function infer](main.py:326) and [Python.function infer_stream](main.py:375)131 132## Configuration (.env)133 134See [.env.example](.env.example)135- PORT (default 3000)136- MODEL_REPO_ID (default "unsloth/Qwen3-4B-Instruct-2507")137- HF_TOKEN (optional)138- MAX_TOKENS (default 256)139- TEMPERATURE (default 0.7)140- MAX_VIDEO_FRAMES (default 16)141- DEVICE_MAP (default "auto")142- TORCH_DTYPE (default "auto")143- PERSIST_SESSIONS (default 0; set 1 to enable SQLite persistence)144- SESSIONS_DB_PATH (default sessions.db)145- SESSIONS_TTL_SECONDS (default 600)146- CANCEL_AFTER_DISCONNECT_SECONDS (default 3600; set 0 to disable)147 148## Error handling and readiness149 150- Health endpoint: [Python.app.get()](main.py:577)151  - Returns { ok, modelReady, modelId, error }152- Chat endpoint:153  - 400 for invalid messages or multimodal parsing errors154  - 503 when model failed to load155  - 500 for unexpected generation errors156- During first request, the model is lazily loaded; subsequent requests reuse the singleton157 158## Performance and scaling159 160- GPU recommended:161  - Set DEVICE_MAP=auto and TORCH_DTYPE=bfloat16/float16 if supported162- Reduce MAX_VIDEO_FRAMES to speed up video processing163- For concurrency:164  - FastAPI/Uvicorn workers and model sharing: typically 1 model per process165  - For high throughput, prefer multiple processes or a queueing layer166 167## Data and directories168 169- models/ contains downloaded model artifacts (implicitly created by Transformers cache); ignored by git170- tmp/ used transiently for video decoding (temporary files)171 172Ignored artifacts (see [.gitignore](.gitignore))173- Python: .venv/, __pycache__/, .cache/, etc.174- Large artifacts: models/, data/, uploads/, tmp/175 176## Streaming resume details177 178- Session store:179  - In-memory ring buffer for fast replay180  - Optional SQLite persistence for robust replay across process restarts181  - See GC in [Python.class _SessionStore](main.py:449) and [Python.method _SQLiteStore.gc](main.py:526)182- Limits:183  - Ring buffer stores ~2048 SSE events per session by default184  - If the buffer overflows before a client resumes and persistence is disabled, the earliest chunks may be unavailable185- End-of-stream:186  - Final chunk contains finish_reason: "stop"187  - "[DONE]" sentinel is emitted afterwards188 189## Marketplace Integration Plan190 191This inference server is designed to power an AI-powered marketplace platform. The following components need to be developed:192 193### 1. Database Schema (Planned)194- **Suppliers table**: id, name, business_name, location (lat/lng), address, contact, registration_date195- **Products table**: id, supplier_id, name, description, price, stock_quantity, category, tags196- **Users table**: id, name, email, location (lat/lng), ai_access_enabled, preferences197- **Conversations table**: id, user_id, session_id, created_at (for chat history)198- **Messages table**: id, conversation_id, role (user/assistant), content, timestamp199 200### 2. Marketplace API Endpoints (Planned)201- **Supplier Management**:202  - POST /api/suppliers/register - Register new supplier203  - POST /api/products - Add product listing204  - PUT /api/products/{id} - Update product (stock, price)205  - GET /api/suppliers/{id}/products - List supplier products206 207- **Product Search**:208  - GET /api/products/search?q={query}&location={lat,lng} - Traditional search209  - POST /api/products/ai-search - AI-powered search with natural language210 211- **AI Assistant Integration**:212  - POST /api/chat - Wrapper around /v1/chat/completions with context injection213  - Context injection: Pass product database results as system message214  - Location awareness: Calculate distance, sort by proximity215  - Example flow:216    1. User: "laptop gaming Jakarta budget 10 juta"217    2. Backend queries products table for: category="laptop", tags LIKE "%gaming%", location near Jakarta, price <= 10000000218    3. Inject results into system prompt: "Available products: [JSON array of matching products]"219    4. Send to /v1/chat/completions220    5. AI recommends from available inventory with reasons221 222### 3. Location-Aware Features (Planned)223- Geolocation distance calculation (Haversine formula)224- Sort products/suppliers by distance from user225- Multi-supplier comparison showing nearest options226- Delivery time estimates based on distance227 228### 4. Context Management Strategy229- Maintain conversation history per user session230- Inject product catalog context dynamically based on query231- Use session_id for resumable conversations232- Store conversation in database for analytics and personalization233 234### Current Status235- ✅ Inference server ready (/v1/chat/completions endpoint)236- ✅ Streaming and resume functionality237- ⏳ Database schema design238- ⏳ Marketplace API development239- ⏳ Frontend/UI development240- ⏳ Product catalog seeding and testing241 242## Future enhancements243 244- Redis persistence:245  - Add a Redis-backed store as a drop-in alongside SQLite246- Token accounting:247  - Populate usage prompt/completion/total tokens when model exposes tokenization costs248- Logging/observability:249  - Structured logs, request IDs, and metrics250 251## Migration notes252 253### From Node.js to Python (2025-10-23)254- All Node.js server files and scripts were removed (index.js, package*.json, scripts/)255- Migrated to Python FastAPI + Transformers stack256- The API remains OpenAI-compatible on /v1/chat/completions with resumable SSE and optional SQLite persistence257 258### From Multimodal to Text-Only (2025-11-13)259- Migrated from Qwen/Qwen3-VL-2B-Thinking (multimodal) to unsloth/Qwen3-4B-Instruct-2507 (text-only)260- **Deprecated features**: KTP OCR endpoint, image/video processing261- **New focus**: AI-powered marketplace intelligence system262- Multimodal code remains in codebase but is non-functional with current model263