KillerKing93/Transformers-InferenceServer-OpenAPI

1# Project Rules and Workflow (Python FastAPI + Transformers)2 3These rules are binding for every change. Keep code, docs, and behavior synchronized at all times.4 5Files referenced below:6- [README.md](README.md)7- [ARCHITECTURE.md](ARCHITECTURE.md)8- [TODO.md](TODO.md)9- [CLAUDE.md](CLAUDE.md)10- [.env.example](.env.example)11- [.gitignore](.gitignore)12- [requirements.txt](requirements.txt)13- [Python.main()](main.py:1)14 15## 1) Documentation rules (must-do on every change)16 17Always update documentation when code or behavior changes.18 19Minimum documentation checklist:20- What changed and where (filenames, sections, or callable links like [Python.function chat_completions()](main.py:591)).21- Why the change was made (problem or requirement).22- How to operate or verify (commands, endpoints, examples).23- Follow-ups or known limitations.24 25Where to update:26- Operator-facing: [README.md](README.md)27- Developer-facing: [CLAUDE.md](CLAUDE.md) (rationale, alternatives, caveats)28- Architecture or flows: [ARCHITECTURE.md](ARCHITECTURE.md)29- Tasks and statuses: [TODO.md](TODO.md)30 31Never skip documentation. If a change is reverted, document the revert.32 33## 2) Git discipline (mandatory)34 35- Always use Git. Every change or progress step MUST be committed and pushed.36  - Windows CMD example:37    - git add .38    - git commit -m "type(scope): short description"39    - git push40- No exceptions. If no remote exists, commit locally and configure a remote as soon as possible. Record any temporary push limitations in [README.md](README.md) and [CLAUDE.md](CLAUDE.md), but commits are still required locally.41- Commit style:42  - Conventional types: chore, docs, feat, fix, refactor, perf, test, build, ci43  - Keep commits small and atomic (one concern per commit).44  - Reference important files in the commit body, for example: updated [Python.function chat_completions()](main.py:591), [README.md](README.md).45- After updating code or docs, commit immediately. Do not batch unrelated changes.46 47## 2.1) Progress log (mandatory)48 49- Every commit MUST include a corresponding entry in [CLAUDE.md](CLAUDE.md) under a “Progress Log” section.50- Each entry must include:51  - Date/time (Asia/Jakarta)52  - Scope and short summary of the change53  - The final Git commit hash and commit message54  - Files and exact callable anchors touched (use clickable anchors), e.g. [Python.function chat_completions()](main.py:591), [README.md](README.md:1), [ARCHITECTURE.md](ARCHITECTURE.md:1)55  - Verification steps and results (curl examples, expected vs actual, notes)56- Required sequence:57  1) Make code changes58  2) Update docs: [README.md](README.md), [ARCHITECTURE.md](ARCHITECTURE.md), [TODO.md](TODO.md), and add a new progress log entry in [CLAUDE.md](CLAUDE.md)59  3) Run Git commands:60     - git add .61     - git commit -m "type(scope): short description"62     - git push63  4) Append the final commit hash to the [CLAUDE.md](CLAUDE.md) entry if it was not known at authoring time64- No code change may land without a synchronized progress log entry.65 66## 3) Large artifacts policy (.gitignore)67 68Never commit large/generated artifacts. Keep the repository lean and reproducible.69 70Must be ignored:71- models/ (downloaded by HF/Transformers cache or tools at runtime)72- .venv/, venv/73- __pycache__/74- .cache/75- uploads/, data/, tmp/76 77See [.gitignore](.gitignore) and extend as needed for new generated outputs. If you add ignores, document the rationale in [CLAUDE.md](CLAUDE.md).78 79## 4) Model policy (Hugging Face / Transformers)80 81Target default model:82- unsloth/Qwen3-4B-Instruct-2507 (Transformers; instruct model).83 84Rules:85- Use Hugging Face Transformers (AutoModelForCausalLM + AutoProcessor) with trust_remote_code=True.86- Do not commit model weights or caches. Let from_pretrained() download to local caches.87- Handle authentication for gated models via HF_TOKEN in [.env.example](.env.example).88- The server must remain OpenAI-compatible at /v1/chat/completions and support multimodal inputs (text, images, videos).89- Keep configuration via environment variables (see [Python.os.getenv()](main.py:67)).90 91## 5) API contract92 93Provide an OpenAI-compatible endpoint:94- POST /v1/chat/completions95 96Minimum behavior:97- Accept model and messages per OpenAI schema (we honor messages; model is informational since server is pinned via env).98- Non-streaming JSON response.99- Streaming SSE response when body.stream=true:100  - Emit OpenAI-style chat.completion.chunk deltas.101  - Include SSE id lines "session_id:index" to support resume via Last-Event-ID.102 103Resume semantics:104- Client provides a session_id (or server generates one).105- Client may reconnect and send Last-Event-ID: session_id:index to replay missed chunks.106- Session data can be persisted (SQLite) if enabled.107 108Manual cancel (custom extension):109- POST /v1/cancel/{session_id} cancels a streaming generation.110- Note: Not part of legacy OpenAI Chat Completions spec. It mirrors the spirit of the newer OpenAI Responses API cancel endpoint.111 112All endpoints must validate inputs, handle timeouts/failures, and return structured JSON errors.113 114## 6) Streaming, persistence, and cancellation115 116- Streaming is implemented via SSE in [Python.function chat_completions()](main.py:591) with token iteration in [Python.function infer_stream](main.py:375).117- In-memory ring buffer per session and optional SQLite persistence for replay across restarts:118  - In-memory: [Python.class _SSESession](main.py:435), [Python.class _SessionStore](main.py:449)119  - SQLite: [Python.class _SQLiteStore](main.py:482) (enabled with PERSIST_SESSIONS=1)120- Resume:121  - Uses SSE id "session_id:index" and Last-Event-ID header (or ?last_event_id=...).122- Auto-cancel on disconnect:123  - If all clients disconnect, generation is cancelled after CANCEL_AFTER_DISCONNECT_SECONDS (default 3600 sec). Configurable via env.124  - Cooperative stop via StoppingCriteria in [Python.function infer_stream](main.py:375).125- Manual cancel:126  - [Python.function cancel_session](main.py:792) to stop a session on demand.127 128## 7) Logging and error handling129 130- Log key lifecycle stages (startup, model load, stream start/stop, resume).131- Redact sensitive fields (e.g., tokens, credentials).132- User errors → 400; model-not-ready → 503; unexpected failures → 500.133- Optionally add structured logging and request IDs in a follow-up.134 135## 8) Architecture documentation136 137Keep [ARCHITECTURE.md](ARCHITECTURE.md) authoritative for:138- Startup flow and lazy model load139- Multimodal preprocessing (images/videos)140- Streaming, resume, persistence, and cancellation flows141- Error/timeout handling142- Extensibility (persistence strategies, cancellation hooks, scaling patterns)143 144Update when code paths or data flows change.145 146## 9) TODO hygiene147 148Track all planned work in [TODO.md](TODO.md):149- Update statuses immediately when tasks start/complete.150- Add newly discovered tasks as soon as they are identified.151- Keep TODO focused, scoped, and prioritized.152 153## 10) Operational requirements and environment154 155Required:156- Python: >= 3.10157- pip158- PyTorch: install a wheel matching platform/CUDA (see [requirements.txt](requirements.txt) notes)159 160Recommended:161- GPU with sufficient VRAM for the chosen model162- Windows 11 supported; Linux/macOS should also work163 164Environment variables (see [.env.example](.env.example)):165- PORT=3000166- MODEL_REPO_ID=unsloth/Qwen3-4B-Instruct-2507167- HF_TOKEN=168- MAX_TOKENS=256169- TEMPERATURE=0.7170- MAX_VIDEO_FRAMES=16171- DEVICE_MAP=auto172- TORCH_DTYPE=auto173- PERSIST_SESSIONS=1|0, SESSIONS_DB_PATH, SESSIONS_TTL_SECONDS174- CANCEL_AFTER_DISCONNECT_SECONDS=3600 (0 to disable)175 176## 11) File responsibilities overview177 178- Server: [Python.main()](main.py:1)179  - API routing, model singleton, inference, streaming, resume, cancel180- Docs: [README.md](README.md), [ARCHITECTURE.md](ARCHITECTURE.md)181- Dev log: [CLAUDE.md](CLAUDE.md)182- Tasks: [TODO.md](TODO.md)183- Config template: [.env.example](.env.example)184- Dependencies: [requirements.txt](requirements.txt)185- Ignores: [.gitignore](.gitignore)186 187## 12) Workflow example (single iteration)188 1891) Make a small, isolated change (e.g., enable SQLite persistence).1902) Update docs:191   - [CLAUDE.md](CLAUDE.md): what/why/how192   - [README.md](README.md): operator usage changes193   - [ARCHITECTURE.md](ARCHITECTURE.md): persistence/resume flow194   - [TODO.md](TODO.md): status changes1953) Commit and push:196   - git add .197   - git commit -m "feat(stream): add SQLite persistence for SSE resume"198   - git push1994) Verify locally; record any issues or follow-ups in [CLAUDE.md](CLAUDE.md).200 201## 13) Compliance checklist (pre-merge / pre-push)202 203- Code runs locally (uvicorn main:app …).204- Docs updated ([README.md](README.md), [CLAUDE.md](CLAUDE.md), [ARCHITECTURE.md](ARCHITECTURE.md), [TODO.md](TODO.md)).205- No large artifacts added to git.206- Commit message follows conventional style.207- Endpoint contract honored (including streaming/resume semantics and cancel extension).208