WaveCut/Bonsai-Chat-WebGPU
Bonsai WebGPU Chat
Private-by-construction, on-device chat with Prism ML's end-to-end 1-bit Bonsai models. Model weights are downloaded directly into browser storage; prompt processing and generation run inside this tab through wllama and llama.cpp WebAssembly/WebGPU.
Runtime tiers
- Bonsai 1.7B — fastest download and broadest CPU fallback.
- Bonsai 4B — balanced local chat.
- Bonsai 8B — primary agentic/tool model.
- Bonsai 27B — WebGPU-only hybrid model with strict adapter and full-graph checks; the UI refuses silent CPU fallback.
The chat UI defaults both context allocation and maximum completion to 8,192 tokens. Prompt text, system instructions, tool schemas, prior turns, and generated tokens share that same context window, so the completion setting is a ceiling rather than a guarantee after a non-empty prompt. Context and completion controls use each model's declared context maximum: 32K for 1.7B and 4B, 64K for 8B, and 262K for 27B.
Wllama allocates the selected KV-cache capacity while loading the model rather than growing it token by token. If that allocation exceeds the browser's available memory, the UI keeps the page alive, unloads the invalid model state, and asks the user to retry with a smaller context. A full context during generation stops with a separate capacity explanation; a WebGPU out-of-memory device loss also invalidates the model and returns the same reduce-context steer. Model loading includes llama.cpp's empty-run warmup. Chrome's JSPI WebGPU path then runs a long prefill followed by a short prompt-cache replacement pass so both batch shapes and decode kernels are compiled before the first user turn; the warmup uses that chat's current system instructions so prompt-cache reuse matches the real conversation. Changing those instructions invalidates the loaded configuration and asks for an explicit reload. Safari's compatibility path keeps only the short native warmup. All preparation remains inside the single annotated load progress bar. If a new prompt shape still requires a kernel, the request reports a live Preparing WebGPU timer before normal prefill telemetry begins.
The Space manifest pins `WaveCut/Bonsai-web-GGUF` to release commit d85382fa09fe868c0242d81488dfc2edd8d3729b (tag v1.1.3); its published release manifest separately pins the immutable weight commit 112ea7a1a6229bde132b176b9a72477a7ecfde64. Every shard has a recorded byte size and SHA-256.
The repository includes the verified dist/ production bundle. The CPU Basic Docker Space serves that bundle directly with Nginx and does not run npm or Vite during its container build. The checked-in bundle is produced locally by npm ci && npm run build; source, tests, lockfile, and exact runtime assets remain in the same Space repository.
Chat workspace
Chat sessions, system instructions, and tool-call logs are saved locally in IndexedDB. The chat list can reopen, continue, or delete a session without touching the model cache. Older records rebuild their tool log from the persisted inline calls. New chats begin with the bundled private-assistant instructions; an empty chat exposes them directly, while an active chat keeps them in Chat settings. Appearance can follow the operating system or be pinned to the light or dark theme. Both sidebars can collapse, and the transcript can switch between centered and full-width layouts.
Every user and assistant message exposes copy and branch-aware regenerate controls. Regenerating a user turn discards the later local branch and answers that turn again; regenerating an assistant turn replaces that response and its later branch. A sent user turn can also be edited in place; saving the edit discards its later branch and immediately generates a replacement answer. Assistant replies preserve reasoning, text, tool calls, structured results, and artifacts as one chronological transcript across repeated agent rounds. Fenced code is highlighted incrementally with the pinned Shiki streaming tokenizer, including incomplete fences while generation is active.
Each artifact is one persistent, versioned web deployment rather than one HTML string. Its browser-ready HTML, CSS, JavaScript, SVG, JSON, and other text files live in a private OPFS workspace; the conversation also keeps a recovery snapshot, so reopening or switching chats restores the matching emulator. The file browser and rendered page stay inline in the transcript, while the right sidebar is only an index that scrolls to them. A successful test opens Rendered automatically. A failed test stays on Files with console diagnostics for the next agent round. Updating an older deployment preserves its stable ID and moves the current artifact card to the new tool-call position. Every artifact card can download its complete nested file tree as a local ZIP.
Workspaces currently accept up to 48 text files, 512 KiB per file, and 2 MiB total. They are for browser-ready sites: there is no npm install, build daemon, server process, WebContainer, binary-file writer, or backend.
Browser requirements
Chrome or Edge with WebGPU is recommended. Safari uses the stable compatibility runtime but is substantially slower, so the UI calls this limitation out before the model is loaded. CPU-WASM fallback requires a cross-origin-isolated browser with SharedArrayBuffer; the Nginx configuration enables it. Model storage uses the browser cache/OPFS path exposed by wllama and requests persistent storage before the first model byte is downloaded. A browser denial is non-fatal, and the storage panel lets the operator request it again.
The Chrome/Edge JSPI release uses the validated raw-layout vec4 Q10 WG64 matvec kernel. Safari keeps the scalar Q10 compat/Asyncify runtime; the two paths have independent artifact hashes and the browser selects between them automatically.
The release manifest still pins every shard SHA-256, but model loading does not scan each multi-gigabyte Blob a second time before handing it to llama.cpp. Fresh downloads must match the pinned byte length; cached shards must also carry the matching pinned SHA-256 in their cache metadata. Invalid or partial shards are removed and retried from byte zero. llama.cpp then performs the mandatory GGUF parse and native/WebGPU load. This avoids a duplicate full-file read, while same-size corruption of an already trusted cache entry may surface only when the native loader consumes it.
27B is intentionally gated. It is unavailable when the adapter's maxStorageBufferBindingSize, shader-f16, browser quota, or all-WebGPU graph requirement does not pass. Chrome on Apple Silicon is the validated release path. Safari 26.5 on Apple Silicon has also passed a real foreground cold 27B load through compat/Asyncify plus explicit OPFS eviction. The normal compat path caps prefill at n_batch=32 and n_ubatch=16 so Safari submits shorter Metal command buffers; explicit benchmark tuning can still override the cap. On Safari 26.5.2, the final 64-token gate completed at 1.2 prefill tok/s and 3.5 decode tok/s with one graph, zero CPU ops, and 65/65 GPU layers. Chrome's JSPI path does not use this cap. Firefox gating was validated on macOS: it selected compat, offered CPU-WASM for dense tiers, and refused 27B without WebGPU; Firefox model generation was not exercised. Windows Chrome/Edge on a discrete GPU and a physical low-limit iGPU laptop remain explicit release-matrix gaps. Dense CPU fallback is offered only when the browser exposes the required cross-origin-isolated WASM features.
Browser benchmark
`?view=bench` is a separate, shareable benchmark surface rendered by the same browser app. The query deep-link keeps the benchmark executable under the same COOP/COEP headers. It loads nothing until Run benchmark is pressed, then records first-load and verified-cache reload wall time, TTFT, prefill/decode throughput, graph placement, device limits, engine revisions, the exact runtime artifact hash, and a fixed temperature-0 prompt. Schema v3 exports the exact messages, sampling controls, raw generated text, pinned model source/shard hashes, and an opt-in sampled-token trace. Normal chat does not request logprobs and returns no token-id trace.
The default field-core-v1 workload remains short. The separate state-drift-1k-v1 evidence workload locks greedy sampling and refuses to export unless all 1,024 sampled token IDs and the 64/128/256/512/768/1,024 uint32-LE prefix hashes are present. Its diagnostic-only 27B WebGPU path also validates the pinned native CPU prompt/reference fixture, then teacher-forces that fixed 1,024-token sequence through raw token-ID prefixes. Schema v3 keeps the forced reference logprob, natural top five, mean NLL, and perplexity separate from the natural browser generation. This scoring method is not available to normal chat.
On the pinned Apple metal-3 validation path, the current JSPI 1,024-token 27B run reached 12.1 tok/s with one graph, zero CPU ops, and 65/65 GPU layers. The exact-current scalar control reached 10.25 tok/s. Both arms returned the same 1,024 token IDs, the same top-five candidate IDs at every position, and the same visible text; the promoted vec4 arm improved decode by 18.05%. Exact CPU/browser token parity remains NO-GO because the first natural mismatch is a reciprocal 13/198 top-one/top-two flip at position 30. The separate numerical gate passes: browser and native CPU mean NLL are 0.0456365662 and 0.0455182718 (absolute delta 0.0001182944 against a 0.01 limit), with near-tie margins 0.006769 and 0.003605. This supports a bounded GPU floating-point tolerance; it is not a bit-exact parity claim. That run uses the exact JSPI and patch hashes published below; the release does not claim bit-exact browser/native parity.
n_batch, n_ubatch, Flash Attention, paired K/V cache types, and the pinned JSPI/compat WASM flavor are available there as explicitly experimental controls. The browser report separates requested and native-observed values, records the actually loaded artifact hashes, and fails the run when they disagree. A pinned local 1.7B WebGPU sweep proved Flash off/auto with F16, Q80, and Q40 KV: every row retained one graph, 29/29 GPU layers, zero CPU ops, and the same short deterministic output hash. Observed KV allocation fell from 112 MiB for F16 to 59.5 MiB for Q80 and 31.5 MiB for Q40.
The separate final-artifact 27B >8K smoke used an 8,448-token context with 8,314 prompt tokens, Flash auto, and Q40/Q40 KV. It measured 7.55 prefill tok/s and 6.42 decode tok/s, retained one graph with zero CPU ops and 65/65 GPU layers, and exported all 8 requested top-five token records. This is one local Apple metal-3 result; it does not establish a cross-device default.
A same-Chrome A/B with one warmup and three alternating measured runs per arm also proved explicit JSPI/compat switching. Median warm load was 2,257.7 ms for JSPI and 2,799.1 ms for compat; all eight outputs and graph tripwires matched. These are experimental single-device results, not release recommendations; normal chat keeps Flash off, F16/F16, and automatic WASM selection.
The historical public v1.1.2 prewarmed-cache baseline below is one Apple metal-3 / headless Chrome sample, not a cross-device study and not a first-download claim:
Local tools and safety boundary
Tools are off by default. Memory writes/deletes require operator confirmation. js_eval runs in a disposable opaque-origin Worker with network and browser storage blocked, a 5-second timeout, and bounded output. The bounded agent loop returns every tool result to the model and keeps each call and result inline at its actual position in the answer. artifact_deploy creates a file tree, artifact_write adds/replaces/deletes files, artifact_inspect reads its manifest or selected files, and artifact_test returns fresh console and DOM diagnostics. The same stable deployment ID is used across repair rounds.
OPFS is the editable source of truth. Each version is also copied to an immutable Cache Storage namespace served by a Service Worker, but cached HTML is deliberately inert (script-src 'none'). Hugging Face Spaces cannot give each deployment its own origin, and an executable same-origin Service Worker URL would make allow-same-origin unsafe. The executable preview therefore uses a stricter in-frame VFS: CSS and local assets are resolved from the snapshot, ES modules use an injected local import map, and relative fetch() reads through a read-only VFS response shim.
That runner is a credentialless opaque-origin allow-scripts iframe without allow-same-origin. A strict CSP blocks external network, storage, workers, frames, forms, popups, and navigation; runtime messages are accepted only from the exact iframe window with serialized origin null and the current deployment token. Console and DOM diagnostics return to the agent loop. The browser does not provide a process watchdog for an opaque srcdoc frame, so deliberately non-yielding synchronous JavaScript can still stall the current tab until it is closed; it does not gain parent, storage, credential, or external-network access. Prompt, token, tool, artifact, and memory data stay in the browser; model/static asset downloads contact Hugging Face, and an approved web_search call sends its query to Wikipedia.
Provenance
- llama.cpp baseline:
00fa7cb284cbf133fc426733bd64238a3588a33e - Dawn native WebGPU validation:
18eb229ef5f707c1464cc581252e7603c73a3ef0 - custom wllama source:
912c18b75d4358c1405a64646b8dbe43a205943b - custom nested llama.cpp:
00fa7cb284cbf133fc426733bd64238a3588a33e - custom patch set:
60a5280bea5fc5ae11065f4c83fd2d532526cdcb7977cbe927cac7ae90926bde - custom ESM:
a02f02b86d7dac8e3bd6c21225910c755bb85ec64ac7d323bfbfa2477b225e3d - JSPI WASM:
dbded8334921b51877ade32009bcaee5b7423e3b39679089a970062c57128acb - compat WASM:
f9b1bd2bd2a62a6b95ab52ecda92d10c709bc1ac3b6be9bb450e5adeff8dd39f - compat worker:
0b667db536815fb9e5dc9ffbfbd6ba29affdda38a4ba892d5372ddb41ce9a8c8 - JSPI Q10 profile: raw-layout vec4 WG64 with shape-aware 4/8/16-output tiling, direct GDN cache writes, and dual FFN projection + SwiGLU fusion; Safari compat Q10 profile: scalar/conservative
The vendored runtime exposes the validated token-embedding WebGPU placement needed for single-graph Bonsai inference and fails closed when placement or the backend tripwire disagrees. Its exact JS/WASM hashes are checked during postinstall; source/patch provenance is in vendor/wllama-bonsai/SOURCE.json. Third-party notices are in THIRD_PARTY_NOTICES.md and public/licenses/.
The pinned WebGPU source implements Flash Attention with Q40/Q80 K/V. The benchmark lane has short 1.7B deterministic-output and memory evidence for both quantized types, a 27B long-state result only for Flash-off F16, and a 27B >8K smoke only for Flash-auto Q40. No 27B Q80 long-state result is claimed. Repeated and cross-device gates remain open, so Flash Attention remains off and F16/F16 remains the release default.
Licenses and attribution
The Bonsai GGUF tiers are Apache-2.0 model artifacts; their upstream notices are published in WaveCut/Bonsai-web-GGUF@v1.1.3/notices/. The shipped wllama/llama.cpp engines, React runtime, @noble/hashes, Shiki, and @shikijs/stream are MIT-licensed. Exact runtime license texts are included in public/licenses/; the consolidated inventory is THIRD_PARTY_NOTICES.md.