borkiss/falcon-perception-webgpu-demo

Falcon-Perception · WebGPU video demo

tiiuae/Falcon-Perception (0.6B open-vocabulary detection VLM) running its full language stack on hand-written WGSL kernels — no ONNX Runtime, no WASM BLAS. Video is served by keyframe detection + a GPU correlation tracker in between, with lossless speculative decoding of the keyframe token pattern drafted from tracker state.

On an Apple M4: full detection query 1.2 s (vs 7.8–8.7 s for the fp32 ONNX/WASM original), decode 91–95 tok/s, keyframes 2.2–2.5 Hz on sparse scenes, tracker 30 FPS, 724 MB download (vs 2.4 GB).

Weights stream from borkiss/falcon-perception-webgpu (FPQ4 container: layer matrices int8, lm_head int4, f16 decode heads) and are cached by the browser. Needs WebGPU with shader-f16 + subgroups (Chrome/Edge 125+).

Drop in a video file, use the webcam, or hit «демо-клип» for a built-in synthetic pan. LIVE mode races the stream in real time; RECORD mode processes a file offline frame-by-frame at max resolution, then plays it back with exact boxes.

There is also a benchmark page — it runs the standard perf suite (load, full photo query, video keyframes with speculative decode, tracker) on your GPU and produces a shareable .log.