apssouza22/webgpu-cluster

WebGPU Cluster

A distributed inference grid that turns browsers with WebGPU into cluster nodes. Host RF-DETR object detection or SmolVLM image description in a Web Worker; a Node broker queues tasks and exposes them over HTTP so any client (curl, Python, Node, etc.) can call your GPU.

Repository: github.com/apssouza22/webgpu-video-cluster Live demo (Hugging Face Space): apssouza22-webgpu-cluster.hf.space · Space repo

curl / Python / app  →  Node broker (task queue)  →  SSE  →  browser host (WebGPU)
                                                              ↓
                                                    RF-DETR · SmolVLM

Inference runs in the host’s browser on their hardware — not on the broker machine. The broker only coordinates tasks and fetches remote images.

Quick start

npm install
npm run dev

1. 1.Open http://localhost:5180 (landing page)
2. 2.Click Join the grid (or open http://localhost:5180/host.html), choose a model to share, pick a host id (e.g. my-gpu-node), and click Start hosting (loads the model on WebGPU; keep the tab open)
3. 3.Open http://localhost:5180/monitor.html to see registered hosts and copy curl examples
4. 4.From another terminal:

curl -X POST 'http://localhost:5180/v1/detect' \
  -H 'Content-Type: application/json' \
  -d '{
    "host": "my-gpu-node",
    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
    "threshold": 0.5
  }'

Response:

{
  "task_id": "...",
  "host": "my-gpu-node",
  "threshold": 0.5,
  "detections": [
    { "label": "cat", "score": 0.92, "box": { "xmin": 10, "ymin": 20, "xmax": 200, "ymax": 180 } }
  ]
}

npm run dev runs Vite (port 5180) and the API broker (port 8787). Vite proxies /v1, /api, and /health to the broker.

Endpoints

`POST /v1/detect` body:

• —host (required) — id from the browser host page
• —image_url or image_base64 (required) — broker fetches URLs server-side
• —threshold (optional, default 0.5)

`POST /v1/describe` body:

• —host (required)
• —image_url or image_base64 (required)
• —instruction (optional, default "What do you see?")
• —max_new_tokens (optional, default 100)

Scripts

Cluster models

Models are defined in shared/clusterModels.ts. The host page dropdown and GET /v1/models both read from that list. To add a model: add an entry there, wire loading in src/pages/addNode.ts (ensureModelLoaded), add a task handler under src/tasks/, and register broker routes in server/.

• —RF-DETR: onnx-community/rfdetr_medium-ONNX via @huggingface/transformers with device: 'webgpu'
• —SmolVLM: HuggingFaceTB/SmolVLM-500M-Instruct with quantized vision/decoder weights (same approach as the SmolVLM realtime WebGPU demo)

Models download from Hugging Face on first load. Inference runs in a Web Worker; the broker sends base64 images and the host converts them to VideoFrame for the worker.

Requirements

• —Browser with WebGPU (Chrome or Edge desktop recommended)
• —Dev server COOP/COEP headers in vite.config.ts (required for Transformers.js / WASM)

Project layout

Hugging Face Space

Hosted at [apssouza22/webgpu-cluster](https://huggingface.co/spaces/apssouza22/webgpu-cluster) (Docker SDK, port 7860).

1. 1.Open the host UI in Chrome or Edge (WebGPU required).
2. 2.Choose a host id and model, then click Start hosting — keep the tab open.
3. 3.Call the API on the same origin:

curl -X POST 'https://apssouza22-webgpu-cluster.hf.space/v1/detect' \
  -H 'Content-Type: application/json' \
  -d '{
    "host": "my-gpu-node",
    "image_url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png",
    "threshold": 0.5
  }'

The Space container serves the broker and static files only; inference runs in the visitor’s browser.

Deploy a new version

npm run build:space
hf upload apssouza22/webgpu-cluster . . \
  --repo-type space \
  --exclude ".git/*" \
  --exclude "node_modules/*" \
  --commit-message "Your change summary"

Wait until the Space shows Running, then check curl https://apssouza22-webgpu-cluster.hf.space/health. Full steps: SPACES.md.

License

MIT