apssouza22/webgpu-cluster

1# WebGPU + AI in the Browser: Real-Time Object Detection on Video2 3*Part 3 of the WebGPU Video Processing series — builds on [Part 2: Hardware-Accelerated Video Processing](https://github.com/apssouza22/webgpu-video-ai/blob/main/ARTICLE.md)*4 5> **Live demo:** [apssouza22.github.io/webgpu-video-ai](https://apssouza22.github.io/webgpu-video-ai) — open it in Chrome or Edge with a GPU, no install required.6> **Source code:** [github.com/apssouza22/webgpu-video-ai](https://github.com/apssouza22/webgpu-video-ai)7 8---9 10In Part 2 we built a complete encode/decode pipeline: MediaBunny decoded video frames, WebGPU composited layers, and WebCodecs exported a final MP4 — all without a server. That pipeline processed video offline, one frame at a time, as fast as the CPU allowed.11 12Part 3 adds a harder constraint: **real-time**. We want to run an AI object detection model on every video frame as it plays and draw bounding boxes on the GPU — without ever stalling the video. That means three things must happen concurrently: decoding, inference, and rendering. This article explains the architecture that makes that possible and why each piece of the stack was chosen.13 14---15 16## Why AI in the browser?17 18Running models in the browser used to mean a Python server, a REST API, and a round-trip that made real-time interaction impractical. Three recent additions change that:19 20- **WebGPU** exposes the GPU's thousands of cores directly to web code, making matrix math fast enough for neural networks.21- **ONNX Runtime Web** compiles a standard model format to run on that GPU from JavaScript.22- **Transformers.js** wraps ONNX Runtime with a Hugging Face–compatible API, so loading a model is one `pipeline()` call.23 24Together they let you download a model once to the browser cache and run inference locally, frame after frame, with no server. You can see this running right now at the [live demo](https://apssouza22.github.io/webgpu-video-ai) — the model downloads on first load, then detects objects in the video entirely on your local GPU.25 26---27 28## System architecture overview29 30Before diving into each component, here is how all the pieces fit together. Three execution contexts run concurrently and communicate through a small set of typed messages and GPU command queues:31 32```mermaid33flowchart TD34    subgraph MT["MAIN THREAD"]35        CP["CompositionPlayer - RAF loop 60fps"]36        VP["VideoPlayer"]37        MB["MediaBunny decoder"]38        GC["GpuCompositor"]39        GVR["GpuVideoRenderer - pass 1"]40        GIR["GpuImageRenderer - pass 2"]41        GDR["GpuDetectionRenderer - pass 3"]42        VOD["VideoObjectDetection"]43        AP["AudioPlayer - Web Audio API"]44        CV["Canvas output"]45    end46 47    subgraph DW["DETECTION WORKER"]48        DWT["detection.worker.ts"]49        CIB["createImageBitmap resize"]50        OC["OffscreenCanvas 576x576"]51        TJ["Transformers.js pipeline()"]52        ONNX["ONNX Runtime Web"]53    end54 55    subgraph GPU["GPU HARDWARE"]56        RQ["Render command queue - video and detection WGSL"]57        CQ["Compute command queue - ONNX inference WGSL"]58    end59 60    CP --> VP61    CP --> AP62    VP --> MB63    MB -->|VideoFrame| GC64    GC --> GVR65    GC --> GIR66    GC --> GDR67    GVR --> CV68    GIR --> CV69    GDR --> CV70    GDR --> RQ71    VP --> VOD72    VOD -->|"postMessage(frame, [frame]) zero-copy"| DWT73    DWT --> CIB --> OC --> TJ --> ONNX --> CQ74    DWT -->|"detect-result bounding boxes"| VOD75    VOD -->|detections| GDR76```77 78The main thread never blocks on inference. The GPU hardware executes both rendering and inference command buffers, potentially in parallel on separate compute units.79 80---81 82## WebGPU: why it matters for AI83 84Graphics and AI share a fundamental operation: multiply a large matrix by a vector, millions of times per second. A CPU executes these sequentially, one element at a time. A GPU executes them in parallel across thousands of shader invocations that all run simultaneously.85 86```mermaid87flowchart LR88    subgraph CPU["CPU - Sequential  O(N^3)"]89        c1["row 0"] --> c2["row 1"] --> c3["row 2"] --> c4["row N"]90    end91    subgraph GPU["GPU - Parallel  O(N) amortized"]92        g1["core 0 - row 0"]93        g2["core 1 - row 1"]94        g3["core 2 - row 2"]95        g4["core N - row N"]96    end97```98 99The difference is not incremental. A modern GPU can sustain hundreds of TFLOPS on matrix operations while a CPU manages a few TFLOPS at best. For a neural network, every layer is a matrix multiply. More parallelism means more layers per second, which means more frames per second.100 101WebGPU exposes this hardware through a low-overhead command model. Instead of issuing GPU calls one by one, you record a `GPUCommandEncoder`, accumulate all operations into it, and submit the whole batch in a single call:102 103```typescript104const encoder = this.device.createCommandEncoder();105const pass = encoder.beginRenderPass({ ... });106pass.setPipeline(this.pipeline);107pass.setBindGroup(0, bindGroup);108pass.draw(6);109pass.end();110this.device.queue.submit([encoder.finish()]);111```112 113The GPU processes the command buffer asynchronously. The CPU continues without waiting. This is the design that makes it possible to both render video and run AI inference at the same time.114 115### Comparison with WebGL116 117WebGL was designed for rendering, not compute. Doing matrix math in WebGL requires abusing fragment shaders — packing data into textures and writing shaders that treat each pixel as a matrix element. It works, but it is awkward, limited in precision, and cannot express modern network architectures cleanly.118 119WebGPU adds a dedicated **compute shader** stage (`@compute`). ONNX Runtime Web uses compute shaders for every operation in a neural network: matrix multiplication, convolution, attention, normalization. The shader code is written in WGSL, the same language used for rendering, and runs on the same hardware. The AI pipeline and the rendering pipeline live in the same GPU process, sharing the same device and queue.120 121---122 123## Transformers.js and ONNX: how they work124 125### ONNX: a universal model format126 127ONNX (Open Neural Network Exchange) is a file format that describes a model as a computation graph. Nodes are operations (matrix multiply, ReLU, softmax), and edges are tensors that flow between them.128 129```mermaid130flowchart TD131    INPUT["Input image - 576x576x3"]132    BACKBONE["Backbone ResNet-D\nConv2D and BatchNorm\nconvolutional feature extraction"]133    ENCODER["Encoder DETR head\nMulti-head self-attention\nMatMul and Softmax and Add"]134    DECODER["Decoder\nCross-attention over object queries"]135    BOX["Box head\nxmin, ymin, xmax, ymax"]136    CLASS["Class head\n60 object labels"]137 138    INPUT --> BACKBONE139    BACKBONE -->|"feature maps"| ENCODER140    ENCODER -->|"encoded tokens"| DECODER141    DECODER --> BOX142    DECODER --> CLASS143```144 145Training frameworks — PyTorch, TensorFlow, JAX — can all export to ONNX. ONNX Runtime then runs that graph on any supported backend: CPU, CUDA, CoreML, or **WebGPU**. When loaded in the browser, ONNX Runtime Web reads the graph, allocates GPU buffers for each tensor, and compiles the operations into WGSL shaders at load time — which is why the first inference ("warmup") always takes longer than subsequent ones.146 147The model used here, `onnx-community/rfdetr_medium-ONNX`, is a real-time detection transformer. It takes a `576×576` image as input and returns bounding boxes with class labels and confidence scores. The ONNX file is hosted on Hugging Face and downloaded into the browser's cache the first time you load the [demo](https://apssouza22.github.io/webgpu-video-ai).148 149### Transformers.js: pipeline abstraction150 151Transformers.js wraps ONNX Runtime Web with a high-level API modeled after the Hugging Face `transformers` Python library. In [`detection.worker.ts`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/detection/detection.worker.ts), loading the model is a single call:152 153```typescript154const MODEL_ID = 'onnx-community/rfdetr_medium-ONNX';155 156detector = await pipeline('object-detection', MODEL_ID, {157  device: 'webgpu',   // compile ONNX graph to WGSL compute shaders158  dtype: 'fp16',      // faster WebGPU inference (fp32 is more accurate)159  progress_callback: (progress) => {160    // streams download progress: file name + percentage161    post({ type: 'status', message: `Downloading ${progress.file} (${percent}%)…` });162  },163});164```165 166`device: 'webgpu'` tells Transformers.js to compile the ONNX graph onto the GPU via WebGPU. The `progress_callback` is useful because the model is tens of megabytes: you can display a real download bar rather than a silent spinner — which is exactly what the [demo](https://apssouza22.github.io/webgpu-video-ai) shows in the status area during first load.167 168After the `await` resolves, `detector` is a callable function. You pass it an image (here an `OffscreenCanvas`) and get back structured results:169 170```typescript171const results = await detect(canvas, { threshold: 0.5, percentage: true });172// [{ label: 'person', score: 0.91, box: { xmin: 0.12, ymin: 0.08, xmax: 0.44, ymax: 0.97 } }, ...]173```174 175`percentage: true` returns box coordinates normalized to 0–1, which matches the coordinate system the WebGPU detection shader uses directly — no conversion needed.176 177---178 179## The parallel processing problem180 181Here is the core tension. Video at 30 fps gives you 33 ms per frame. Inference alone exceeds that budget on most hardware:182 183```mermaid184gantt185    title Per-frame time budget at 30fps (33ms window)186    dateFormat x187    axisFormat %Lms188 189    section Main thread190        Decode frame (MediaBunny)  : 0, 5191        Cache image overlays       : 5, 6192        GPU render 3 passes        : 6, 8193 194    section Worker thread195        AI inference RF-DETR       : crit, 0, 80196```197 198RF-DETR on a mid-range GPU takes 30–80 ms. If inference ran synchronously on the main thread, the video would stutter every single frame.199 200The key insight is that inference and rendering **do not need to happen in the same frame**. The detections from frame N are still valid for frames N+1 and N+2 — objects do not teleport. You can display slightly stale boxes and refresh them as new inference results arrive asynchronously. That observation unlocks the architecture.201 202---203 204## Web Workers: the isolation layer205 206A Web Worker is a JavaScript execution context that runs on a separate OS thread. It shares no memory with the main thread, has its own event loop, and cannot access the DOM. What it *can* access: WebGPU, `OffscreenCanvas`, `VideoFrame`, and `fetch`.207 208Isolating the model inside a Worker gives you three critical properties:209 210**1. The main thread is never blocked.** Model loading, WGSL shader compilation, and inference all happen on the worker thread. The `requestAnimationFrame` loop that drives video playback continues at 60 fps regardless of what the model is doing.211 212**2. The GPU is shared.** Both the worker's WebGPU inference pipeline and the main thread's rendering pipeline target the same physical GPU. They share no JavaScript objects, but the hardware executes their command buffers concurrently when possible.213 214**3. `VideoFrame` transfer is zero-copy.** `VideoFrame` is a transferable object. When you pass it in the `postMessage` transfer list, the underlying GPU texture buffer moves to the worker without copying pixel data:215 216```mermaid217sequenceDiagram218    participant MT as Main Thread219    participant W as Detection Worker220 221    MT->>MT: decode VideoFrame (owns GPU buffer)222    MT->>W: postMessage(frame, [frame]) - zero copy, no memcpy223    Note over MT: frame is neutered, cannot be used224    Note over W: frame received, owns the GPU buffer225    W->>W: createImageBitmap resize to 576x576226    W->>W: ONNX inference on GPU227    W-->>MT: detect-result with bounding boxes228    Note over W: worker calls frame.close()229```230 231A 1280×720 frame at 4 bytes per pixel is 3.5 MB. Copying that on every frame would add ~100 MB/s of memory bandwidth overhead — the zero-copy transfer makes the overhead negligible:232 233```typescript234// ObjectDetector.ts — main thread235this.worker.postMessage(236  { type: 'detect', id, threshold, frame },237  [frame],   // ← transfer list: frame is moved, not copied238);239// frame is now null/neutered on the main thread240```241 242### The typed message protocol243 244The worker communicates through a narrow, explicitly typed protocol defined in [`workerMessages.ts`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/detection/workerMessages.ts):245 246```mermaid247sequenceDiagram248    participant MT as Main Thread249    participant W as Detection Worker250 251    MT->>W: type init252    W-->>MT: type status - Downloading model253    W-->>MT: type status - Compiling RF-DETR shaders254    W-->>MT: type ready255 256    MT->>W: type detect, id 42, threshold 0.5, frame VideoFrame257    Note over W: resize to 576x576, run ONNX graph258    W-->>MT: type detect-result, id 42, results array259 260    MT->>W: type detect, id 43, frame VideoFrame261    W-->>MT: type error, id 43, error message262```263 264The numeric `id` field is essential: it matches each response to its originating request. The `ObjectDetector` class on the main thread stores pending promises in a `Map<id, {resolve, reject}>`:265 266```typescript267// ObjectDetector.ts268async detect(frame: VideoFrame, options: { threshold: number }): Promise<DetectionResult[]> {269  const id = this.nextId++;270  return new Promise((resolve, reject) => {271    this.pending.set(id, { resolve, reject });272    this.worker.postMessage({ type: 'detect', id, threshold: options.threshold, frame }, [frame]);273  });274}275 276// on 'detect-result' message:277const pending = this.pending.get(message.id);278pending?.resolve(message.results);279this.pending.delete(message.id);280```281 282This is a clean bridge from a message-passing interface to async/await without a library.283 284---285 286## The detection pipeline end to end287 288### Step 1 — Frame preprocessing inside the worker289 290RF-DETR expects a `576×576` image. Video frames are typically `1280×720` or larger. The worker resizes the transferred `VideoFrame` using `createImageBitmap` with explicit resize parameters, then draws it onto a persistent `OffscreenCanvas`:291 292```typescript293// detection.worker.ts294const MODEL_INPUT_SIZE = 576;295 296async function runDetection(frame: VideoFrame, threshold: number) {297  const { canvas, ctx } = getPreprocessSurface(); // reused 576×576 OffscreenCanvas298 299  const bitmap = await createImageBitmap(frame, {300    resizeWidth: MODEL_INPUT_SIZE,301    resizeHeight: MODEL_INPUT_SIZE,   // GPU-accelerated resize302  });303  ctx.drawImage(bitmap, 0, 0);304  bitmap.close();305  frame.close(); // release the transferred VideoFrame306 307  return detect(canvas, { threshold, percentage: true });308}309```310 311`createImageBitmap` with resize parameters is GPU-accelerated in browsers that support it. The `OffscreenCanvas` is created once with `{ willReadFrequently: true }` so the browser optimizes its backing store for the one CPU readback that Transformers.js needs to build the input tensor.312 313```mermaid314flowchart TD315    VF["VideoFrame 1280x720\ntransferred from main thread"]316    CIB["createImageBitmap\nresizeWidth 576, resizeHeight 576\nGPU-accelerated downscale"]317    OC["OffscreenCanvas 576x576\nreused each frame, willReadFrequently true"]318    T1["1. Read pixels from canvas"]319    T2["2. Build Float32 input tensor"]320    T3["3. Upload tensor to GPU buffer"]321    T4["4. Run ONNX graph via WGSL compute shaders"]322    T5["5. Readback output tensor to CPU"]323    RES["DetectionResult array\nlabel, score, box coords normalized 0 to 1"]324 325    VF -->|"frame.close() after bitmap"| CIB326    CIB --> OC327    OC --> T1 --> T2 --> T3 --> T4 --> T5 --> RES328```329 330### Step 2 — Throttling: one frame in flight331 332Running inference takes longer than a single video frame. The `VideoObjectDetection` class in [`VideoObjectDetection.ts`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/player/VideoObjectDetection.ts) implements a simple but effective throttle: if a detection is already running, the incoming frame is held as a "pending" replacement. When inference completes, it immediately starts on the pending frame, dropping any intermediate frames:333 334```typescript335schedule(videoFrame: VideoFrame): void {336  if (this.detectionBusy) {337    this.pendingDetectionFrame?.close();            // discard previous pending338    this.pendingDetectionFrame = new VideoFrame(videoFrame);339    return;340  }341  void this.runDetection(new VideoFrame(videoFrame));342}343```344 345The state machine looks like this:346 347```mermaid348flowchart TD349    A["schedule videoFrame"] --> B{detectionBusy?}350    B -- NO --> C["runDetection frame\ndetectionBusy = true"]351    B -- YES --> D["pendingFrame = frame\nclose previous pending if any"]352    C --> E["await ONNX inference"]353    E --> F["store detections\ndetectionBusy = false"]354    F --> G{pendingFrame?}355    G -- YES --> H["runDetection pendingFrame\npendingFrame = null"]356    G -- NO --> I["idle - await next schedule"]357    H --> E358    D -.->|waits implicitly| F359```360 361At any given moment there is at most one detection in flight and one pending. The detection rate tracks hardware capability automatically — a fast GPU detects every frame, a slow GPU skips frames without any explicit rate-limiting code.362 363### Step 3 — Version-gating stale results364 365There is a subtle race: between submitting a detection and receiving the result, the user might seek to a different position in the video. Applying old bounding boxes to new content looks wrong. An incrementing version counter handles this:366 367```typescript368private async runDetection(detectionFrame: VideoFrame): Promise<void> {369  const detectionVersion = ++this.detectionVersion;370  // ...inference (may take 50ms)...371  if (detectionVersion !== this.detectionVersion) {372    return; // a newer detection was started — discard this result373  }374  this.detections = toGpuDetections(results);375}376```377 378Every call to `runDetection` increments the counter. On completion, it checks that the counter has not moved. If it has, the stale result is silently dropped.379 380---381 382## Rendering detections on the GPU383 384The bounding boxes returned by the model are JavaScript objects. To draw them without looping over pixels in JavaScript, they are packed into a WebGPU uniform buffer and sent to a WGSL shader that draws the boxes entirely on the GPU.385 386### The uniform buffer layout387 388```389Offset 0  ┌─────────────────────────────────┐390          │  count    (u32,  4 bytes)        │391Offset 4  │  lineWidth (f32, 4 bytes)        │392Offset 8  │  _pad      (vec2f, 8 bytes)      │393Offset 16 ├─────────────────────────────────┤  ← boxes start394          │  box[0].bounds (vec4f, 16 bytes) │  xmin, ymin, xmax, ymax395          │  box[0].color  (vec4f, 16 bytes) │  r, g, b, alpha396Offset 48 ├─────────────────────────────────┤397          │  box[1].bounds (16 bytes)        │398          │  box[1].color  (16 bytes)        │399          ├─────────────────────────────────┤400          │  ...                             │401          │  box[31]                         │402Offset 1040└─────────────────────────────────┘403Total: 16 + 32 × 32 = 1,040 bytes (fixed, allocated once at startup)404```405 406Fixed at 32 boxes maximum — enough for any real scene, and the constant size means the buffer is allocated once at startup and reused every frame with a single `writeBuffer` call.407 408### The fragment shader draws borders analytically409 410Instead of uploading one quad per bounding box (which would require a dynamic vertex buffer), the shader uses a **full-screen triangle pass** and tests each pixel analytically against all boxes in the uniform buffer:411 412```wgsl413fn onBorder(uv: vec2f, bounds: vec4f, lineWidth: f32) -> bool {414  let insideX = step(bounds.x, uv.x) * step(uv.x, bounds.z);415  let insideY = step(bounds.y, uv.y) * step(uv.y, bounds.w);416 417  let nearLeft   = abs(uv.x - bounds.x) < lineWidth;418  let nearRight  = abs(uv.x - bounds.z) < lineWidth;419  let nearTop    = abs(uv.y - bounds.y) < lineWidth;420  let nearBottom = abs(uv.y - bounds.w) < lineWidth;421 422  return (nearLeft  && insideY > 0.5) ||423         (nearRight && insideY > 0.5) ||424         (nearTop   && insideX > 0.5) ||425         (nearBottom && insideX > 0.5);426}427 428@fragment429fn fragmentMain(input: VertexOutput) -> @location(0) vec4f {430  var color = vec4f(0.0);  // transparent by default431  for (var i = 0u; i < detection.count; i++) {432    let box = detection.boxes[i];433    if (onBorder(input.uv, box.bounds, detection.lineWidth)) {434      color = vec4f(box.color.rgb, 1.0);435    }436  }437  return color;438}439```440 441Every fragment (pixel) independently tests whether it falls on the border of any detection box. The GPU runs these tests in parallel across all pixels simultaneously — ~921,600 pixels × 5 box tests = ~4.6M comparisons completed in a single render pass in under 1ms. No geometry uploads, no dynamic buffers, no JavaScript loops over pixels.442 443### The compositing order444 445The [`GpuCompositor`](https://github.com/apssouza22/webgpu-video-ai/blob/main/src/gpu/GpuCompositor.ts) issues three render passes per frame in a fixed order, all targeting the same canvas texture:446 447```mermaid448flowchart TD449    CT["Canvas texture 1280x720"]450    P1["Pass 1 - GpuVideoRenderer\nloadOp clear\nVideoFrame via importExternalTexture\nalpha blend"]451    S1["video frame rendered"]452    P2["Pass 2 - GpuImageRenderer\nloadOp load\nHTMLImageElement to rgba8unorm cached texture\nalpha blend"]453    S2["video plus image overlays"]454    P3["Pass 3 - GpuDetectionRenderer\nloadOp load\nuniform buffer with box list\nalpha blend"]455    OUT["Final composited frame\nvideo, overlays, and detection boxes"]456 457    CT --> P1 --> S1 --> P2 --> S2 --> P3 --> OUT458```459 460Each pass's `loadOp: 'load'` means "start from what the previous pass wrote." WebGPU serializes passes within a queue submission, so each pass sees the accumulated result of all prior passes. The detection layer always renders last, so boxes are always visible on top of any overlay.461 462---463 464## The warmup problem465 466ONNX Runtime Web compiles WGSL shaders for each operation in the model graph on first use. On a complex model like RF-DETR, this takes one to three seconds and produces a visible freeze if it happens during playback. The fix is one inference before the user presses play:467 468```typescript469// main.ts470setStatus('Warming up RF-DETR shaders (first inference)…');471await player.getVideoPlayer().warmupDetection(0);472```473 474This decodes frame 0, runs it through the full detection pipeline, and discards the result. The side effect is that all WGSL shaders are compiled and cached by the WebGPU driver. Every subsequent inference hits already-compiled pipelines and runs at steady-state speed. You can observe the warmup message in the status bar when you first open the [demo](https://apssouza22.github.io/webgpu-video-ai).475 476---477 478## Required HTTP headers479 480Transformers.js uses `SharedArrayBuffer` internally for efficient WASM memory management. `SharedArrayBuffer` is gated behind cross-origin isolation — the page must be served with:481 482```483Cross-Origin-Opener-Policy: same-origin484Cross-Origin-Embedder-Policy: require-corp485```486 487Without them, `SharedArrayBuffer` is unavailable and Transformers.js either falls back to a slower code path or fails outright. In Vite, these are set in the dev server configuration:488 489```typescript490// vite.config.ts491server: {492  headers: {493    'Cross-Origin-Opener-Policy': 'same-origin',494    'Cross-Origin-Embedder-Policy': 'require-corp',495  },496}497```498 499You must also set them in production. A missing COEP header is one of the most common reasons a Transformers.js app works in development but silently fails after deployment. Check them first when debugging.500 501---502 503## Key lessons504 505- **Transfer, do not copy.** `VideoFrame` is a transferable. Pass it with `[frame]` in the `postMessage` transfer list to move the GPU buffer to the worker with zero memory copy. Copying 3.5 MB of pixel data per frame destroys throughput.506 507- **Warm up before you need it.** ONNX shader compilation happens on first inference. Run one dummy call at startup — the [demo](https://apssouza22.github.io/webgpu-video-ai) does this before revealing the play button — so playback never triggers a compile stall mid-video.508 509- **One detection in flight.** Queuing detections faster than the model can process them causes memory pressure and bursty latency. A single pending-frame slot gives you the freshest possible result at any hardware speed, without any explicit rate-limiting.510 511- **Version-gate results.** Any async operation that feeds the render loop needs a staleness check. An incrementing counter is enough. Without it, seeking causes stale boxes to flash on screen.512 513- **Analyze pixels in the shader, not JavaScript.** Drawing bounding boxes as a full-screen analytical test is faster than uploading per-box geometry because the GPU runs the test across all pixels simultaneously with zero JavaScript overhead per pixel.514 515- **COEP/COOP are not optional.** Set them in both development and production. Check them first when debugging a Transformers.js deployment.516 517---518 519## What this architecture enables520 521The pipeline described here runs a 60-class object detector on live video at whatever rate the hardware supports — from a few detections per second on integrated graphics to near-frame-rate on a discrete GPU — without ever stalling video playback. You can see the detection FPS counter in the [live demo](https://apssouza22.github.io/webgpu-video-ai) updating in real time as the model runs.522 523More broadly, the same structure applies to any model you want to run on video: pose estimation, segmentation, depth prediction, face recognition. The worker isolation, the zero-copy transfer, the warmup, and the GPU rendering layer are all model-agnostic. Swap the Transformers.js `pipeline()` call and update the WGSL uniform struct, and the architecture carries the new task without structural changes.524 525The browser is no longer a thin client that sends video to a server for processing. It is a capable ML runtime with direct access to the GPU, a mature async concurrency model, and zero-copy primitives that make real-time inference practical. The gap between what you can build in the browser and what requires a backend is narrowing fast.526 527---528 529*Full source code: [github.com/apssouza22/webgpu-video-ai](https://github.com/apssouza22/webgpu-video-ai)*530*Live demo: [apssouza22.github.io/webgpu-video-ai](https://apssouza22.github.io/webgpu-video-ai)*