macmacmacmac/qwen-webgpu-lora

1---2license: apache-2.03language:4- en5base_model:6- WeiboAI/VibeThinker-3B7pipeline_tag: text-generation8tags:9- webgpu10- in-browser11- lora12- client-side13- edge14- qwen2.515library_name: emberglass16---17 18<h1 align="center">🜂 EMBERGLASS</h1>19<p align="center"><em>A 3-billion-parameter mind, running inside a browser tab. No server. No install. No upload. Just a page.</em></p>20 21<p align="center"><b>~35 tokens/sec decode · live LoRA hot-swap · bit-exact to the reference · 100% client-side WebGPU</b></p>22 23> **Code & runtime:** https://github.com/maceip/emberglass24 25---26 27## What this is28 29Most "AI in the browser" is a thin client phoning home to someone else's GPU. **This isn't that.**30 31Emberglass is a hand-built inference engine that runs a fine-tuned **Qwen2.5-3B** reasoning model **entirely on your own machine's GPU**, from inside a single static web page — written from the metal up in raw WebGPU compute shaders. The model thinks for thousands of tokens, streams a verdict, and **never sends a single byte off your device.** You bring the weights; the page brings the engine.32 33And the part that shouldn't be possible at this speed: you can **swap the model's personality at runtime.** Load the base once, then hot-swap LoRA adapters *live* — no reload, no recompile, no re-quantization. The base weights never move. The output changes the instant you flip the adapter, and flips back **bit-for-bit identically** when you remove it.34 35## Why it's hard (and why it's fast)36 37A browser tab is the most hostile environment imaginable for a 3B-parameter model. No CUDA. No vendor kernels. A 5.4 GB weight shard won't even fit in a single JavaScript array. Every fast path that exists on a server is closed. So we closed the gap by hand:38 39- **Custom WGSL compute kernels** for every op — the only way LoRA could become live and swappable instead of a baked-in constant.40- **int4 group-128 quantization** that is **numerically exact** on the reference decode — half the memory, zero quality lost.41- **Split-K flash-style decode attention** so it stays fast even at thousands of tokens of context.42- **Subgroup-reduction GEMV** + a **GPU-resident batched decode loop** (argmax→embed stays on the GPU; one sync per batch).43 44Every win was found by **measuring** — nanosecond GPU timestamp profiling — not guessing. 9 → 35 tok/s over one focused push.45 46## Results47 48| | |49|---|---|50| Decode speed | ~35 tok/s across a full multi-thousand-token reasoning generation |51| Correctness | argmax + every generated token **exact** vs the HuggingFace reference; bit-exact run-to-run |52| LoRA hot-swap | load base once · swap live · perfect restore on clear · no reload |53| Footprint | one static HTML page; weights supplied by the visitor (BYO-model) |54| Privacy | absolute — inference never leaves the device |55 56## Context window & prefill sizes57 58The base model — [WeiboAI/VibeThinker-3B](https://huggingface.co/WeiboAI/VibeThinker-3B), a Qwen2.5-architecture 3B reasoning model (from Qwen2.5-Coder-3B) — supports **131072 (128K) positions** with a **32K sliding window**, and is built to *think long*: its generation config defaults to `max_new_tokens=65536`, and the authors suggest **60K–100K tokens** for the hardest problems. So context length is a first-class concern here, not an afterthought.59 60The runtime exposes context + prefill as options:61 62```js63const rt = new QwenWGPU(device, QWEN25_3B, { maxCtx: 8192, maxPrefillT: 8192 });64```65 66- **`maxCtx`** — the context window (KV-cache length). Decode attention is **split-K** and prefill attention is **flash / online-softmax** (O(block) workgroup memory, not O(ctx)), so neither caps out at small sizes — context scales until you run out of VRAM.67- **`maxPrefillT`** — the largest prompt processed in one batched (tiled-int4-GEMM) prefill pass. Longer prompts (or prefill while a LoRA adapter is active) fall back to the sequential path; clamped to `maxCtx`.68 69Defaults are **8192 / 8192** — ample for the bug-bounty triage adapter (its chain-of-thought runs a few thousand tokens) at a modest footprint. Raise them toward the base model's 128K as memory allows. **The KV cache is the cost**, and it grows linearly (~72 KB per token of context, f32, across all 36 layers):70 71| context (`maxCtx`) | KV cache (f32) |72|---|---|73| 8 192 *(default)* | ~0.6 GB |74| 16 384 | ~1.2 GB |75| 32 768 *(sliding window)* | ~2.4 GB |76| 131 072 *(max positions)* | ~9.4 GB |77 78Plus ~2 GB of int4/int8 weights and lazily-sized prefill scratch. **Verified in-browser:** batched prefill is bit-exact to the sequential path through ctx 1024; runs end-to-end at 4 096 / 8 192; and a `maxCtx: 16384` build prefills a 9 000-token prompt and decodes past it. (KV is f32 today — quantizing it would roughly halve these numbers.)79 80## Note on weights81 82**This page hosts no multi-GB weights.** Emberglass is the *engine*; it is bring-your-own-model. Point it at a Qwen2.5-3B (or compatible) checkpoint served locally and it quantizes to int4 on the way to the GPU. Drag in a PEFT/MLX LoRA adapter to hot-swap a specialization live.83 84## Run it85 86See **https://github.com/maceip/emberglass**. Requires a WebGPU browser exposing the `subgroups` feature. Built and validated on an Apple M5 Max.87 88---89<p align="center"><sub>Built the hard way, on purpose. 🜂</sub></p>90