apssouza22/webgpu-cluster

1---2title: GPU Detection Cluster3emoji: 🎮4colorFrom: blue5colorTo: purple6sdk: docker7app_port: 78608pinned: false9models:10  - onnx-community/rfdetr_medium-ONNX11  - HuggingFaceTB/SmolVLM-500M-Instruct12---13 14# WebGPU Cluster15 16A **distributed inference grid** that turns browsers with WebGPU into cluster nodes. Host **RF-DETR** object detection or **SmolVLM** image description in a Web Worker; a **Node broker** queues tasks and exposes them over HTTP so any client (`curl`, Python, Node, etc.) can call your GPU.17 18**Repository:** [github.com/apssouza22/webgpu-video-cluster](https://github.com/apssouza22/webgpu-video-cluster)  19**Live demo (Hugging Face Space):** [apssouza22-webgpu-cluster.hf.space](https://apssouza22-webgpu-cluster.hf.space/) · [Space repo](https://huggingface.co/spaces/apssouza22/webgpu-cluster)20 21```text22curl / Python / app  →  Node broker (task queue)  →  SSE  →  browser host (WebGPU)23                                                              ↓24                                                    RF-DETR · SmolVLM25```26 27Inference runs **in the host’s browser** on their hardware — not on the broker machine. The broker only coordinates tasks and fetches remote images.28 29## Quick start30 31```bash32npm install33npm run dev34```35 361. Open **http://localhost:5180** (landing page)372. Click **Join the grid** (or open **http://localhost:5180/host.html**), choose a **model to share**, pick a host id (e.g. `my-gpu-node`), and click **Start hosting** (loads the model on WebGPU; keep the tab open)383. Open **http://localhost:5180/monitor.html** to see registered hosts and copy curl examples394. From another terminal:40 41```bash42curl -X POST 'http://localhost:5180/v1/detect' \43  -H 'Content-Type: application/json' \44  -d '{45    "host": "my-gpu-node",46    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",47    "threshold": 0.548  }'49```50 51Response:52 53```json54{55  "task_id": "...",56  "host": "my-gpu-node",57  "threshold": 0.5,58  "detections": [59    { "label": "cat", "score": 0.92, "box": { "xmin": 10, "ymin": 20, "xmax": 200, "ymax": 180 } }60  ]61}62```63 64`npm run dev` runs **Vite** (port 5180) and the **API broker** (port 8787). Vite proxies `/v1`, `/api`, and `/health` to the broker.65 66## Endpoints67 68| Method | Path | Description |69|--------|------|-------------|70| `POST` | `/v1/detect` | Object detection with RF-DETR (waits for host) |71| `POST` | `/v1/describe` | Image description with SmolVLM (waits for host) |72| `GET` | `/v1/hosts` | List registered hosts and online status |73| `GET` | `/v1/models` | Cluster models (id, label, implementation status) |74| `GET` | `/v1/tasks/:id` | Task status / results |75| `GET` | `/health` | Broker health check |76| — | `/` | Landing page — join or view the grid |77| — | `/host.html` | Browser host — register and share GPU |78| — | `/monitor.html` | Dashboard — live host list and curl examples |79| `POST` | `/api/hosts/register` | Called by the browser host |80| `GET` | `/api/hosts/stream?host_id=` | SSE — browser host receives tasks |81 82**`POST /v1/detect` body:**83 84- `host` (required) — id from the browser host page85- `image_url` or `image_base64` (required) — broker fetches URLs server-side86- `threshold` (optional, default `0.5`)87 88**`POST /v1/describe` body:**89 90- `host` (required)91- `image_url` or `image_base64` (required)92- `instruction` (optional, default `"What do you see?"`)93- `max_new_tokens` (optional, default `100`)94 95## Scripts96 97| Command | Description |98|---------|-------------|99| `npm run dev` | Vite (5180) + API broker (8787) |100| `npm run dev:api` | API broker only |101| `npm run dev:web` | Vite only (proxies API when broker is running) |102| `npm run build` | Typecheck + production bundle to `docs/` (base `/webgpu-video-ai/` for GitHub Pages) |103| `npm run build:space` | Same bundle with base `/` for Hugging Face Spaces |104| `npm run preview` | Serve the production build locally |105| `npm run start` | Broker + static UI (`SERVE_STATIC=1`, `PORT` default 8787) |106| `npm run start:api` | Run broker without watch |107 108## Cluster models109 110Models are defined in `shared/clusterModels.ts`. The host page dropdown and `GET /v1/models` both read from that list. To add a model: add an entry there, wire loading in `src/pages/addNode.ts` (`ensureModelLoaded`), add a task handler under `src/tasks/`, and register broker routes in `server/`.111 112| Model id | Endpoint | Worker |113|----------|----------|--------|114| `rfdetr-medium` | `POST /v1/detect` | `src/detection/detection.worker.ts` |115| `smolvlm-500m` | `POST /v1/describe` | `src/videodescription/videodescription.worker.ts` |116 117- **RF-DETR:** `onnx-community/rfdetr_medium-ONNX` via `@huggingface/transformers` with `device: 'webgpu'`118- **SmolVLM:** `HuggingFaceTB/SmolVLM-500M-Instruct` with quantized vision/decoder weights (same approach as the [SmolVLM realtime WebGPU demo](https://huggingface.co/spaces/webml-community/smolvlm-realtime-webgpu))119 120Models download from Hugging Face on first load. Inference runs in a **Web Worker**; the broker sends base64 images and the host converts them to `VideoFrame` for the worker.121 122## Requirements123 124- Browser with **WebGPU** (Chrome or Edge desktop recommended)125- Dev server COOP/COEP headers in `vite.config.ts` (required for Transformers.js / WASM)126 127## Project layout128 129| Path | Role |130|------|------|131| `server/` | Express broker — task queue, host registry, SSE |132| `src/pages/addNode.ts` | Browser host — register, pull tasks, run inference |133| `src/pages/clusterMonitor.ts` | Monitor dashboard |134| `src/detection/` | RF-DETR worker and main-thread API |135| `src/videodescription/` | SmolVLM worker and main-thread API |136| `shared/clusterModels.ts` | Model catalog for UI and API |137 138## Hugging Face Space139 140Hosted at **[apssouza22/webgpu-cluster](https://huggingface.co/spaces/apssouza22/webgpu-cluster)** (Docker SDK, port 7860).141 142| | URL |143|--|-----|144| Landing | [apssouza22-webgpu-cluster.hf.space](https://apssouza22-webgpu-cluster.hf.space/) |145| Host UI | [apssouza22-webgpu-cluster.hf.space/host.html](https://apssouza22-webgpu-cluster.hf.space/host.html) |146| Monitor | [apssouza22-webgpu-cluster.hf.space/monitor.html](https://apssouza22-webgpu-cluster.hf.space/monitor.html) |147 1481. Open the host UI in **Chrome or Edge** (WebGPU required).1492. Choose a host id and model, then click **Start hosting** — keep the tab open.1503. Call the API on the same origin:151 152```bash153curl -X POST 'https://apssouza22-webgpu-cluster.hf.space/v1/detect' \154  -H 'Content-Type: application/json' \155  -d '{156    "host": "my-gpu-node",157    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",158    "threshold": 0.5159  }'160```161 162The Space container serves the broker and static files only; **inference runs in the visitor’s browser**.163 164### Deploy a new version165 166```bash167npm run build:space168hf upload apssouza22/webgpu-cluster . . \169  --repo-type space \170  --exclude ".git/*" \171  --exclude "node_modules/*" \172  --commit-message "Your change summary"173```174 175Wait until the Space shows **Running**, then check `curl https://apssouza22-webgpu-cluster.hf.space/health`. Full steps: [SPACES.md](./SPACES.md#deploy-a-new-version).176 177## License178 179MIT180