jeanr2025/local-ai

Eric

Eric is a static multimodal chat app. Chat inference, image understanding, image generation, speech recognition, and speech synthesis run in the visitor's browser—there is no application backend, account, API key, analytics service, or paid Space hardware.

The interface loads the published theme and input component from statistikZH/leu. AI execution stays in dedicated Web Workers so the composer, attachments, language selector, and background downloads remain usable while a model is loading or answering.

The compact chat-model picker sits inside the composer immediately before Send. Choose a model and send normally: if that model is not in memory, Eric loads it, preserves the draft and attachments, and submits the queued message automatically when the model is ready. Completed history survives the switch. Precision, device, response length, manual preload/reload, background caching, runtime diagnostics, and cache clearing live in the Advanced drawer opened from the top-left; the drawer is collapsed by default on desktop and mobile.

Transformers.js is pinned to browser release 4.2.0, which includes both Qwen 3.5 and Gemma 4. CDN modules are imported lazily inside their workers so a failed dependency reports through the app instead of preventing the worker from starting.

Chat models

The two Qwen 3.5 entries use the optimized ONNX repositories and component-level q4/q4f16 loading. Gemma 4 and Qwen 3.5 use their native multimodal processor classes. Adding an image selects Gemma 4 automatically when the current model is text-only; an already-loaded Qwen 3.5 model can accept it directly.

Only one chat runtime is kept in inference memory. Completed conversation history survives a model switch. Up to two additional model workers can download and warm a model in the background, then immediately dispose it while retaining its files in the browser cache. This makes concurrent caching possible, but it can use substantial network bandwidth, RAM, and GPU memory while the workers initialize.

Image generation

Open the toolbox beside the attachment + and select Generate an image. Submitted prompts run SD-Turbo at 512 × 512. The image runtime stays loaded for consecutive prompts, avoiding model reloads; deselect the tool when you want Eric to release it and restore the previous chat model. Generated images are displayed in the conversation with a PNG download link, but they are not inserted into the language-model history.

The implementation follows Microsoft's ONNX Runtime Web SD-Turbo reference: ONNX Runtime Web 1.17.1, WebGPU/fp16, a CLIP tokenizer and text encoder, one U-Net evaluation, and the VAE decoder. It uses the same `schmuell/sd-turbo-ort-web` ONNX conversion. The three ONNX files total approximately 2.4 GB (1700 MB text encoder, 640 MB U-Net, 95 MB VAE), so “Turbo” describes its one-step inference—not a small first download.

SD-Turbo image generation is available only when the browser exposes WebGPU and shader-f16; there is no WASM fallback. Before loading it, Eric terminates the active chat worker and pauses background chat-model prefetch workers so the large inference runtimes are never resident together. SD-Turbo sessions remain in the image worker until the tool is deselected. Eric then restores the saved chat configuration from browser cache and resumes paused downloads.

The ONNX mirror is labeled Stability AI non-commercial. Review its linked license before publishing a commercial deployment; the app exposes the feature but does not broaden the model license.

Speech

• —Whisper begins loading automatically in a background worker after the browser runtime check. Its status is shown separately in the Runtime panel, and chat-model loading remains independent.
• —Speech-to-text uses multilingual `onnx-community/whisper-base`, with a q8 encoder and q4 decoder. The DE/EN selector provides the transcription language and one model handles both languages.
• —Text-to-speech uses the browser's installed German or English system voice. It starts immediately, adds no model download, and can be cancelled synchronously for voice-mode interruption. Markdown, links, list markers, code fences, and other display formatting are converted to plain speech text first.
• —The microphone records for at most 60 seconds. Transcribed text is placed in the composer for review; it is not sent automatically.
• —The speaker button at the far right of the composer opens a consent notice and starts voice conversation mode. The normal composer is locked while the mode is active. Tap the large red recording control to finish a turn; Eric transcribes it, loads the selected model if necessary, and answers automatically. Complete answer sentences are queued to the system voice as they stream, so playback can start before the LLM finishes the full response. Tap the central control while Eric is answering to stop playback, interrupt generation without unloading the chat model, and immediately start recording the next turn. Reasoning text and partial control markers are never spoken. Tap the persistent composer speaker button again to leave voice mode at any time.

Cache controls

“Clear local model cache” asks for confirmation, stops active workers, and clears Cache Storage, IndexedDB, and Origin Private File System data for this site. That includes chat models, SD-Turbo, Whisper, and any other app-origin cached data. A page reload follows, and all models must be downloaded again.

Run locally

Serve the directory over HTTP. Module workers and CDN module imports do not run from file://.

python3 -m http.server 8765

Then open <http://localhost:8765> in current Chrome or Edge. Microphone access works on localhost. A basic Python server does not send cross-origin isolation headers, so WebGPU works but multithreaded WASM may not.

Run source-level tests with:

node --test tests/*.test.mjs

Publish on Hugging Face Spaces

Create a Static Space and upload this directory. The YAML block above selects index.html and configures COOP/COEP headers. No Space CPU or GPU upgrade is needed because every model runs on the visitor's hardware.

After publishing, check the browser console:

crossOriginIsolated === true

Also test one chat model, one image-understanding request with Gemma/Qwen 3.5, two consecutive SD-Turbo generations without a reload, deselection and chat-model restoration, microphone permission, DE/EN system speech, voice interruption, and the cache-clear confirmation on the deployed origin. External CDN/model requests must keep working under cross-origin isolation.

Runtime notes

• —WebGPU is the intended runtime. WASM is a slower fallback and can be impractical for the larger chat models.
• —Model sizes are estimates and can change when upstream ONNX repositories are updated.
• —Qwen models use sampling. DeepSeek/Qwen reasoning is separated from the final response, and only the cleaned final response enters chat history.
• —Stream rendering removes Gemma, Llama, Qwen, DeepSeek, ChatML, and slash-turn control markers, including markers split across token chunks.
• —Assistant responses render safe local Markdown, including headings, lists, tables, blockquotes, links, emphasis, and fenced code. Raw HTML and unsafe link protocols are escaped.
• —Uploaded images remain local. This version accepts PNG, JPEG, WebP, or GIF files up to 10 MB each, with four images per message.