KillerKing93/Transformers-InferenceServer-OpenAPI
0
1---2title: "Transformers Inference Server (Qwen3‑VL)"3emoji: 🐍4colorFrom: purple5colorTo: green6sdk: docker7app_port: 30008pinned: false9---10 11# Python FastAPI Inference Server (OpenAI-Compatible) for Qwen3-4B-Instruct12 13## AI-Powered Marketplace Intelligence System14 15This repository provides an OpenAI-compatible inference server powered by Qwen3-4B-Instruct, designed to serve as the AI backend for a **smart marketplace platform** where:16 17- **Suppliers** can register and list their products18- **Users** can query product availability, get AI-powered recommendations, and find the nearest suppliers19- **AI Assistant** helps users discover products based on their needs, location, and preferences20 21The system has been migrated from a Node.js/llama.cpp stack to a Python/Transformers stack for better model compatibility and performance.22 23Key files:24 25- Server entry: [main.py](main.py)26- Environment template: [.env.example](.env.example)27- Python dependencies: [requirements.txt](requirements.txt)28- Architecture: [ARCHITECTURE.md](ARCHITECTURE.md)29 30Model:31 32- Default: unsloth/Qwen3-4B-Instruct-2507 (Transformers; text-only instruct model)33- You can change the model via environment variable MODEL_REPO_ID.34 35## Marketplace Features (Planned)36 37This inference server will power the following marketplace capabilities:38 39### 1. Supplier Management40- Suppliers can register their business41- List products with details (name, price, stock, location)42- Update inventory in real-time43 44### 2. AI-Powered Product Discovery45Users with AI access can:46- **Query Stock Availability**: "Apakah ada stok laptop gaming di Jakarta?"47- **Get Recommendations**: "Saya butuh laptop untuk programming, budget 10 juta, yang bagus apa?"48- **Find Nearest Suppliers**: Products are matched based on user's location to show the closest available suppliers49- **Compare Products**: AI helps compare features, prices, and reviews50 51### 3. Location-Aware Intelligence52- Automatic supplier-to-user distance calculation53- Prioritize recommendations based on proximity54- Suggest delivery or pickup options based on distance55 56### 4. Natural Language Interaction57- Users can ask in natural Indonesian or English58- AI understands context and user preferences over conversation59- Personalized recommendations based on chat history60 61### Current Status62✅ **FULLY IMPLEMENTED!** All marketplace features are production-ready:63- ✅ Inference server (OpenAI-compatible `/v1/chat/completions`)64- ✅ Marketplace database (5 tables with relationships and indexes)65- ✅ 10+ RESTful API endpoints (suppliers, products, users, AI search)66- ✅ Location-aware search with Haversine distance calculation67- ✅ AI-powered natural language product search68- ✅ Sample data seeding script69- ⏳ Frontend UI (planned)70 71## Marketplace API Endpoints72 73### Supplier Management74 75#### Register Supplier76```bash77POST /api/suppliers/register78Content-Type: application/json79 80{81 "name": "John Doe",82 "business_name": "Tech Store Jakarta",83 "email": "john@techstore.com",84 "phone": "+62812345678",85 "address": "Jl. Sudirman No. 123",86 "latitude": -6.2088,87 "longitude": 106.8456,88 "city": "Jakarta",89 "province": "DKI Jakarta"90}91```92 93#### List Suppliers94```bash95GET /api/suppliers?city=Jakarta&skip=0&limit=10096```97 98#### Get Supplier Details99```bash100GET /api/suppliers/{supplier_id}101```102 103### Product Management104 105#### Create Product106```bash107POST /api/suppliers/{supplier_id}/products108Content-Type: application/json109 110{111 "name": "Laptop ASUS ROG Strix G15",112 "description": "Gaming laptop with RTX 4060",113 "price": 15999000,114 "stock_quantity": 5,115 "category": "laptop",116 "tags": "gaming,asus,rtx",117 "sku": "ASU-ROG-G15-001"118}119```120 121#### Update Product122```bash123PUT /api/products/{product_id}124Content-Type: application/json125 126{127 "price": 14999000,128 "stock_quantity": 3129}130```131 132#### List Products133```bash134GET /api/products?category=laptop&min_price=5000000&max_price=20000000&available_only=true135```136 137#### Search Products (Location-Aware)138```bash139GET /api/products/search?q=laptop+gaming&city=Jakarta&user_lat=-6.2088&user_lon=106.8456&max_price=15000000140```141 142Response includes `distance_km` field for each product (distance from user location).143 144### User Management145 146#### Register User147```bash148POST /api/users/register149Content-Type: application/json150 151{152 "name": "Jane Smith",153 "email": "jane@email.com",154 "phone": "+62856789012",155 "latitude": -6.2088,156 "longitude": 106.8456,157 "city": "Jakarta",158 "province": "DKI Jakarta",159 "ai_access_enabled": true160}161```162 163#### Get User Profile164```bash165GET /api/users/{user_id}166```167 168### AI-Powered Search (Premium Feature)169 170The main feature! Natural language product search with AI recommendations.171 172```bash173POST /api/chat/search174Content-Type: application/json175 176{177 "user_id": 1,178 "query": "Saya butuh laptop gaming di Jakarta, budget 12 juta",179 "session_id": "optional-session-id"180}181```182 183**Response:**184```json185{186 "session_id": "user_1_1731485760",187 "response": "Berdasarkan budget Anda 12 juta dan lokasi di Jakarta, saya merekomendasikan HP Pavilion Gaming (Rp 11.999.000) dari Tech Store Surabaya. Laptop ini memiliki RTX 3050 graphics yang bagus untuk gaming...",188 "products_found": 3,189 "conversation_id": 1190}191```192 193**How it works:**1941. Parses natural language query (extracts category, budget, location)1952. Searches database with filters1963. Sorts results by distance from user1974. Injects product catalog into AI system prompt1985. AI generates personalized recommendation1996. Saves conversation to database for history200 201**Requirements:**202- User must have `ai_access_enabled: true`203- Location coordinates help with distance sorting204 205### Database Setup206 207#### 1. Install Dependencies208```bash209pip install sqlalchemy alembic210```211 212#### 2. Configure Database213Create `.env` file (or use `.env.example`):214```env215DATABASE_URL=sqlite:///./marketplace.db216# Or PostgreSQL: postgresql://user:password@localhost/marketplace217# Or MySQL: mysql+pymysql://user:password@localhost/marketplace218```219 220#### 3. Seed Sample Data221```bash222python seed_data.py223```224 225This creates:226- 5 suppliers (Jakarta, Bandung, Surabaya, Jakarta Selatan, Medan)227- 15 products (laptops, smartphones, monitors, accessories)228- 3 users (2 with AI access enabled)229 230#### 4. Start Server231```bash232python main.py233```234 235Database tables are auto-created on first startup.236 237### Testing Marketplace Endpoints238 239Run comprehensive test suite:240```bash241pytest tests/test_marketplace.py -v242```243 244Tests cover:245- Supplier registration and listing246- Product creation, update, and search247- User registration248- Location-aware search249- AI-powered search (mocked)250- Utility functions (Haversine distance, location parsing)251 252## Deprecated Features253 254**Note**: Previous multimodal features (KTP OCR, image/video processing) are **deprecated** as of the migration to Qwen3-4B-Instruct (text-only model). The `/ktp-ocr/` endpoint code remains but is non-functional with the current model.255 256## Hugging Face Space257 258The project is hosted on Hugging Face Spaces for easy access: [KillerKing93/Transformers-TextEngine-InferenceServer-OpenAPI-Compatible-V3](https://huggingface.co/spaces/KillerKing93/Transformers-TextEngine-InferenceServer-OpenAPI-Compatible-V3)259 260You can use the Space's API endpoints directly or access the web UI.261 262## Quick Start263 264### Option 1: Run with Docker (with-model images: CPU / NVIDIA / AMD)265 266Tags built by CI:267- ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-cpu268- ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-nvidia269- ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-amd270 271Pull:272 273```bash274# CPU275docker pull ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-cpu276 277# NVIDIA (CUDA 12.4 wheel)278docker pull ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-nvidia279 280# AMD (ROCm 6.2 wheel)281docker pull ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-amd282```283 284Run:285 286```bash287# CPU288docker run -p 3000:3000 \289 -e HF_TOKEN=your_hf_token_here \290 ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-cpu291 292# NVIDIA GPU (requires NVIDIA drivers + nvidia-container-toolkit on the host)293docker run --gpus all -p 3000:3000 \294 -e HF_TOKEN=your_hf_token_here \295 ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-nvidia296 297# AMD GPU ROCm (requires ROCm 6.2+ drivers on the host; Linux only)298# Map ROCm devices and video group (may vary by distro)299docker run --device=/dev/kfd --device=/dev/dri --group-add video \300 -p 3000:3000 \301 -e HF_TOKEN=your_hf_token_here \302 ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-amd303```304 305Health check:306```bash307curl http://localhost:3000/health308```309 310Swagger UI:311http://localhost:3000/docs312 313OpenAPI (YAML):314http://localhost:3000/openapi.yaml315 316Notes:317- These are with-model images; the first pull is large. In CI, after "Model downloaded." BuildKit may appear idle while tarring/committing the multi‑GB layer.318- Host requirements:319 - NVIDIA: recent driver + nvidia-container-toolkit.320 - AMD: ROCm 6.2+ driver stack, supported GPU, and mapped /dev/kfd and /dev/dri devices.321 322### Option 2: Run Locally323 324Requirements325 326- Python 3.10+327- pip328- PyTorch (install a wheel matching your platform/CUDA)329- Optionally a GPU with enough VRAM for the chosen model330 331Install332 3331. Create and activate a virtual environment (Windows CMD):334 python -m venv .venv335 .venv\Scripts\activate336 3372. Install dependencies:338 pip install -r requirements.txt339 3403. Install PyTorch appropriate for your platform (examples):341 CPU-only:342 pip install torch --index-url https://download.pytorch.org/whl/cpu343 CUDA 12.4 example:344 pip install torch --index-url https://download.pytorch.org/whl/cu124345 3464. Create a .env from the template and adjust if needed:347 copy .env.example .env348 - Set HF_TOKEN if the model is gated349 - Adjust MAX_TOKENS, TEMPERATURE, DEVICE_MAP, TORCH_DTYPE, MAX_VIDEO_FRAMES as desired350 351Configuration via .env352See [.env.example](.env.example). Important variables:353 354- PORT=3000355- MODEL_REPO_ID=unsloth/Qwen3-4B-Instruct-2507356- HF_TOKEN= # optional if gated357- MAX_TOKENS=4096358- TEMPERATURE=0.7359- MAX_VIDEO_FRAMES=16360- DEVICE_MAP=auto361- TORCH_DTYPE=auto362 363Additional streaming/persistence configuration364 365- PERSIST_SESSIONS=1 # enable SQLite-backed resumable SSE366- SESSIONS_DB_PATH=sessions.db # SQLite db path367- SESSIONS_TTL_SECONDS=600 # TTL for finished sessions before GC368- CANCEL_AFTER_DISCONNECT_SECONDS=3600 # auto-cancel generation if all clients disconnect for this many seconds (0=disable)369 370Cancel session API (custom extension)371 372- Endpoint: POST /v1/cancel/{session_id}373- Purpose: Manually cancel an in-flight streaming generation for the given session_id. Not part of OpenAI Chat Completions spec (the newer OpenAI Responses API has cancel), so this is provided as a practical extension.374- Example (Windows CMD):375 curl -X POST http://localhost:3000/v1/cancel/mysession123376 Run377 378- Direct:379 python main.py380 381- Using uvicorn:382 uvicorn main:app --host 0.0.0.0 --port 3000383 384Endpoints (OpenAI-compatible)385 386- Swagger UI387 GET /docs388- OpenAPI (YAML)389 GET /openapi.yaml390- Health391 GET /health392 Example:393 curl http://localhost:3000/health394 Response:395 {396 "ok": true,397 "modelReady": true,398 "modelId": "unsloth/Qwen3-4B-Instruct-2507",399 "error": null400 }401 402- Chat Completions (non-streaming)403 POST /v1/chat/completions404 Example (Windows CMD):405 curl -X POST http://localhost:3000/v1/chat/completions ^406 -H "Content-Type: application/json" ^407 -d "{\"model\":\"qwen-local\",\"messages\":[{\"role\":\"user\",\"content\":\"Describe this image briefly\"}],\"max_tokens\":4096}"408 409- KTP OCR410 POST /ktp-ocr/411 Example (Windows CMD):412 curl -X POST http://localhost:3000/ktp-ocr/ ^413 -F "image=@image.jpg"414 415 Example (PowerShell):416 $body = @{417 model = "qwen-local"418 messages = @(@{ role = "user"; content = "Hello Qwen3!" })419 max_tokens = 4096420 } | ConvertTo-Json -Depth 5421 curl -Method POST http://localhost:3000/v1/chat/completions -ContentType "application/json" -Body $body422 423- Chat Completions (streaming via Server-Sent Events)424 Set "stream": true to receive partial deltas as they are generated.425 Example (Windows CMD):426 curl -N -H "Content-Type: application/json" ^427 -d "{\"model\":\"qwen-local\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: what is 17 * 23?\"}],\"stream\":true}" ^428 http://localhost:3000/v1/chat/completions429 430 The stream format follows OpenAI-style SSE:431 data: { "id": "...", "object": "chat.completion.chunk", "choices":[{ "delta": {"role": "assistant"} }]}432 data: { "choices":[{ "delta": {"content": "To"} }]}433 data: { "choices":[{ "delta": {"content": " think..."} }]}434 ...435 data: { "choices":[{ "delta": {}, "finish_reason": "stop"}]}436 data: [DONE]437 438Multimodal Usage439 440- Text only:441 { "role": "user", "content": "Summarize: The quick brown fox ..." }442 443- Image by URL:444 {445 "role": "user",446 "content": [447 { "type": "text", "text": "What is in this image?" },448 { "type": "image_url", "image_url": { "url": "https://example.com/cat.jpg" } }449 ]450 }451 452- Image by base64:453 {454 "role": "user",455 "content": [456 { "type": "text", "text": "OCR this." },457 { "type": "input_image", "b64_json": "<base64 of image bytes>" }458 ]459 }460 461- Video by URL (frames are sampled up to MAX_VIDEO_FRAMES):462 {463 "role": "user",464 "content": [465 { "type": "text", "text": "Describe this clip." },466 { "type": "video_url", "video_url": { "url": "https://example.com/clip.mp4" } }467 ]468 }469 470- Video by base64:471 {472 "role": "user",473 "content": [474 { "type": "text", "text": "Count the number of cars." },475 { "type": "input_video", "b64_json": "<base64 of full video file>" }476 ]477 }478 479Implementation Notes480 481- Server code: [main.py](main.py)482 - FastAPI with CORS enabled483 - Non-streaming and streaming endpoints484 - Uses AutoProcessor and AutoModelForCausalLM with trust_remote_code=True485 - Converts OpenAI-style messages into the Qwen multimodal format486 - Images loaded via PIL; videos loaded via imageio.v3 (preferred) or OpenCV as fallback; frames sampled487 488Performance Tips489 490- On GPUs: set DEVICE_MAP=auto and TORCH_DTYPE=bfloat16 or float16 if supported491- Reduce MAX_VIDEO_FRAMES to speed up video processing492- Tune MAX_TOKENS and TEMPERATURE according to your needs493 494Troubleshooting495 496- ImportError or no CUDA found:497 - Ensure PyTorch is installed with the correct wheel for your environment.498- OOM / CUDA out of memory:499 - Use a smaller model, lower MAX_VIDEO_FRAMES, lower MAX_TOKENS, or run on CPU.500- 503 Model not ready:501 - The first request triggers model load; check /health for errors and HF_TOKEN if gated.502 503License504 505- See LICENSE for terms.506 507Changelog and Architecture508 509- See [ARCHITECTURE.md](ARCHITECTURE.md) for detailed technical documentation and [CLAUDE.md](CLAUDE.md) for development progress logs.510 511## Streaming behavior, resume, and reconnections512 513The server streams responses using Server‑Sent Events (SSE) from [Python.function chat_completions()](main.py:457), driven by token iteration in [Python.function infer_stream](main.py:361). It now supports resumable streaming using an in‑memory ring buffer and SSE Last-Event-ID, with optional SQLite persistence (enable PERSIST_SESSIONS=1).514 515What’s implemented516 517- Per-session in-memory ring buffer keyed by session_id (no external storage).518- Each SSE event carries an SSE id line in the format "session_id:index" so clients can resume with Last-Event-ID.519- On reconnect:520 - Provide the same session_id in the request body, and521 - Provide "Last-Event-ID: session_id:index" header (or query ?last_event_id=session_id:index).522 - The server replays cached events after index and continues streaming new tokens.523- Session TTL: ~10 minutes, buffer capacity: ~2048 events. Old or finished sessions are garbage-collected in-memory.524 525How to start a streaming session526 527- Minimal (server generates a session_id internally for SSE id lines):528 Windows CMD:529 curl -N -H "Content-Type: application/json" ^530 -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^531 http://localhost:3000/v1/chat/completions532 533- With explicit session_id (recommended if you want to resume):534 Windows CMD:535 curl -N -H "Content-Type: application/json" ^536 -d "{\"session_id\":\"mysession123\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^537 http://localhost:3000/v1/chat/completions538 539How to resume after disconnect540 541- Use the same session_id and the SSE Last-Event-ID header (or ?last_event_id=...):542 Windows CMD (resume from index 42):543 curl -N -H "Content-Type: application/json" ^544 -H "Last-Event-ID: mysession123:42" ^545 -d "{\"session_id\":\"mysession123\",\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" ^546 http://localhost:3000/v1/chat/completions547 548 Alternatively with query string:549 http://localhost:3000/v1/chat/completions?last_event_id=mysession123:42550 551Event format552 553- Chunks follow the OpenAI-style "chat.completion.chunk" shape in data payloads, plus an SSE id:554 id: mysession123:5555 data: {"id":"mysession123","object":"chat.completion.chunk","created":..., "model":"...", "choices":[{"index":0,"delta":{"content":" token"},"finish_reason":null}]}556 557- The stream ends with:558 data: {"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}559 data: [DONE]560 561Notes and limits562 563- This implementation keeps session state only in memory; restarts will drop buffers.564- If the buffer overflows before you resume, the earliest chunks may be unavailable.565- Cancellation on client disconnect is not automatic; generation runs to completion in the background. A cancellable stopping-criteria can be added if required.566 567## Hugging Face repository files support568 569This server loads the Qwen3 model via Transformers with `trust_remote_code=True`, so the standard files from the repo are supported and consumed automatically. Summary for https://huggingface.co/unsloth/Qwen3-4B-Instruct-2507/tree/main:570 571- Used by model weights and architecture572 573 - model.safetensors — main weights loaded by AutoModelForCausalLM574 - config.json — architecture/config575 - generation_config.json — default gen params (we may override via request or env)576 577- Used by tokenizer578 579 - tokenizer.json — primary tokenizer specification580 - tokenizer_config.json — tokenizer settings581 - merges.txt and vocab.json — fallback/compat files; if tokenizer.json exists, HF generally prefers it582 583- Used by processors (multimodal)584 585 - preprocessor_config.json — image/text processor config586 - video_preprocessor_config.json — video processor config (frame sampling, etc.)587 - chat_template.json — chat formatting used by [Python.function infer](main.py:312) and [Python.function infer_stream](main.py:361) via `processor.apply_chat_template(...)`588 589- Not required for runtime590 - README.md, .gitattributes — ignored by runtime591 592Notes:593 594- We rely on Transformers’ AutoModelForCausalLM and AutoProcessor to resolve and use the above files; no manual parsing is required in our code.595- With `trust_remote_code=True`, model-specific code from the repo may load additional assets transparently.596- If the repo updates configs (e.g., new chat template), the server will pick them up on next load.597 598## Cancellation and session persistence599 600- Auto-cancel on disconnect:601 602 - Generation is automatically cancelled if all clients disconnect for more than CANCEL_AFTER_DISCONNECT_SECONDS (default 3600 seconds = 1 hour). Configure in [.env.example](.env.example) via `CANCEL_AFTER_DISCONNECT_SECONDS`.603 - Implemented by a timer in [Python.function chat_completions](main.py:732) that triggers a cooperative stop through a stopping criteria in [Python.function infer_stream](main.py:375).604 605- Manual cancel API (custom extension):606 607 - Endpoint: `POST /v1/cancel/{session_id}`608 - Cancels an ongoing streaming session and marks it finished in the store. Example (Windows CMD):609 curl -X POST http://localhost:3000/v1/cancel/mysession123610 - This is not part of OpenAI’s legacy Chat Completions spec. OpenAI’s newer Responses API has a cancel endpoint, but Chat Completions does not. We provide this custom endpoint for operational control.611 612- Persistence:613 - Optional SQLite-backed persistence for resumable SSE (enable `PERSIST_SESSIONS=1` in [.env.example](.env.example)).614 - Database path: `SESSIONS_DB_PATH` (default: sessions.db)615 - Session TTL for GC: `SESSIONS_TTL_SECONDS` (default: 600)616 - See implementation in [Python.class \_SQLiteStore](main.py:481) and integration in [Python.function chat_completions](main.py:591).617 - Redis is not implemented yet; the design isolates persistence so a Redis-backed store can be added as a drop-in.618 619## Deploy on Render620 621Render has two easy options. Since our image already bakes the model, the fastest path is to deploy the public Docker image (CPU). Render currently doesn’t provide NVIDIA/AMD GPUs for standard Web Services, so use the CPU image.622 623Option A — Deploy public Docker image (recommended)6241) In Render Dashboard: New → Web Service6252) Environment: Docker → Public Docker image6263) Image627 - ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-cpu6284) Instance and region629 - Region: closest to your users630 - Instance type: pick a plan with at least 16 GB RAM (more if you see OOM)6315) Port/health632 - Render auto-injects PORT; the server binds to it via [Python.os.getenv()](main.py:71)633 - Health Check Path: /health (served by [Python.function health](main.py:871))6346) Start command635 - Leave blank; the image uses CMD ["python","main.py"] as defined in [Dockerfile](Dockerfile:54). The app entry is [Python.main()](main.py:1).6367) Environment variables637 - EAGER_LOAD_MODEL=1638 - MAX_TOKENS=4096639 - HF_TOKEN=your_hf_token_here (only if the model is gated)640 - Optional persistence:641 - PERSIST_SESSIONS=1642 - SESSIONS_DB_PATH=/data/sessions.db (requires a disk)6438) Persistent Disk (optional)644 - Add a Disk (e.g., 1–5 GB) and mount it at /data if you enable SQLite persistence6459) Create Web Service and wait for it to start64610) Verify647 - curl https://YOUR-SERVICE.onrender.com/health648 - OpenAPI YAML: https://YOUR-SERVICE.onrender.com/openapi.yaml (served by [Python.function openapi_yaml](main.py:863))649 - Chat endpoint: POST https://YOUR-SERVICE.onrender.com/v1/chat/completions (implemented in [Python.function chat_completions](main.py:891))650 651Option B — Build directly from this GitHub repo (Dockerfile)6521) In Render Dashboard: New → Web Service → Build from a Git repo (connect this repo)6532) Render will detect the Dockerfile automatically (no Build Command needed)6543) Advanced → Docker Build Args655 - BACKEND=cpu (ensures CPU-only torch wheel)6564) Health and env vars657 - Health Check Path: /health658 - Set EAGER_LOAD_MODEL, MAX_TOKENS, HF_TOKEN as needed (same as Option A)6595) (Optional) Add a Disk and mount at /data, then set SESSIONS_DB_PATH=/data/sessions.db if you want resumable SSE across restarts6606) Deploy (first build can take a while due to the multi-GB model layer)661 662Notes and limits on Render663- GPU acceleration (NVIDIA/AMD) isn’t available for standard Web Services on Render; use the CPU image.664- The image already contains the Qwen3-VL model under /app/hf-cache, so there’s no model download at runtime.665- SSE is supported; streaming is produced by [Python.function chat_completions](main.py:891). Keep the connection open to avoid idle timeouts.666- If you enable SQLite persistence, remember to attach a Disk; otherwise, the DB is ephemeral.667 668Example render.yaml (optional IaC)669If you prefer infrastructure-as-code, you can use a render.yaml like:670 671services:672 - type: web673 name: qwen-vl-cpu674 env: docker675 image:676 url: ghcr.io/killerking93/transformers-inferenceserver-openapi-compatible:latest-with-model-cpu677 plan: standard678 region: oregon679 healthCheckPath: /health680 autoDeploy: true681 envVars:682 - key: EAGER_LOAD_MODEL683 value: "1"684 - key: MAX_TOKENS685 value: "4096"686 # - key: HF_TOKEN687 # sync: false # set in dashboard or use Render secrets688 # - key: PERSIST_SESSIONS689 # value: "1"690 # - key: SESSIONS_DB_PATH691 # value: "/data/sessions.db"692 disks:693 # Uncomment if using persistence694 # - name: data695 # mountPath: /data696 # sizeGB: 5697 698After deploy:699- Health: GET /health700- OpenAPI: GET /openapi.yaml701- Inference:702 curl -X POST https://YOUR-SERVICE.onrender.com/v1/chat/completions \703 -H "Content-Type: application/json" \704 -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Hello\"}],\"max_tokens\":4096}"705 706## Deploy on Hugging Face Spaces707 708Recommended: Docker Space (works with our FastAPI app and preserves multimodal behavior). You can run CPU or GPU hardware. To persist the HF cache across restarts, enable Persistent Storage and point HF cache to /data.709 710A) Create the Space (Docker)7111) Install CLI and login:712 pip install -U "huggingface_hub[cli]"713 huggingface-cli login714 7152) Create a Docker Space (public or private):716 huggingface-cli repo create my-qwen3-vl-server --type space --sdk docker717 7183) Add the Space as a remote and push this repo:719 git remote add hf https://huggingface.co/spaces/YOUR_USERNAME/my-qwen3-vl-server720 git push hf main721 722This pushes Dockerfile, main.py, requirements.txt. The Space will auto-build your container.723 724B) Configure Space settings725- Hardware:726 - CPU: works out-of-the-box (fast to build, slower inference).727 - GPU: choose a GPU tier (e.g., T4/A10G/L4) for faster inference.728 729- Persistent Storage (recommended):730 - Enable Persistent storage (e.g., 10–30 GB).731 - This lets you cache models and sessions across restarts.732 733- Variables and Secrets:734 - Variables:735 - EAGER_LOAD_MODEL=1736 - MAX_TOKENS=4096737 - HF_HOME=/data/hf-cache738 - TRANSFORMERS_CACHE=/data/hf-cache739 - Secrets:740 - HF_TOKEN=your_hf_token_if_model_is_gated741 742C) CPU vs GPU on Spaces743- CPU: No change needed. Our Dockerfile defaults to CPU PyTorch and bakes the model during build. It will run on CPU Spaces.744- GPU: Edit the Space’s Dockerfile to switch the backend before the next build:745 - In the file editor of the Space UI, change:746 ARG BACKEND=cpu747 to:748 ARG BACKEND=nvidia749 - Save/commit; the Space rebuilds with a CUDA-enabled torch. Choose a GPU hardware tier in the Space settings. Note: Building the GPU image pulls CUDA torch wheels and increases build time.750- AMD ROCm is not available on Spaces; use NVIDIA GPUs on Spaces.751 752D) Speed up cold starts and caching753- With Persistent Storage enabled and HF_HOME/TRANSFORMERS_CACHE pointed to /data/hf-cache, the model cache persists across restarts (subsequent spins are much faster).754- Keep the Space “Always on” if available on your plan to avoid cold starts.755 756E) Space endpoints757- Base URL: https://huggingface.co/spaces/YOUR_USERNAME/my-qwen3-vl-server (Spaces proxy to your container)758- Swagger UI: GET /docs (interactive API with examples)759- Health: GET /health (implemented by [Python.function health](main.py:951))760- OpenAPI YAML: GET /openapi.yaml (implemented by [Python.openapi_yaml](main.py:943))761- Chat Completions: POST /v1/chat/completions (non-stream + SSE) [Python.function chat_completions](main.py:971)762- Cancel: POST /v1/cancel/{session_id} [Python.function cancel_session](main.py:1191)763 764F) Quick test after the Space is “Running”765- Health:766 curl -s https://YOUR-SPACE-Subdomain.hf.space/health767- Non-stream:768 curl -s -X POST https://YOUR-SPACE-Subdomain.hf.space/v1/chat/completions \769 -H "Content-Type: application/json" \770 -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Hello from HF Spaces!\"}],\"max_tokens\":4096}"771- Streaming:772 curl -N -H "Content-Type: application/json" \773 -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Think step by step: 17*23?\"}],\"stream\":true}" \774 https://YOUR-SPACE-Subdomain.hf.space/v1/chat/completions775 776Notes777- The Space build step can appear “idle” after “Model downloaded.” while Docker commits a multi‑GB layer; this is expected.778- If you hit OOM, increase the Space hardware memory or switch to a GPU tier. Reduce MAX_VIDEO_FRAMES and MAX_TOKENS if needed.779