Modelpublic

mehmetkeremturkcan/mote

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

0likes

Model Card

Mote: Gemma-4-E2B on a Raspberry Pi 5

Mote is a community project for running capable language models on small, common hardware: the Intelligence of Things. This release runs Gemma-4-E2B, an effective 2.3B-parameter model, at 11 tokens per second on a Raspberry Pi 5, on four Arm Cortex-A76 cores with 8 GB of RAM and no accelerator.

Mote is independent and not affiliated with the developers of Gemma.

—Source, kernels, and full build: https://github.com/mkturkcan/mote
—Project page: https://mkturkcan.github.io/mote

Results

Measured on a Raspberry Pi 5 with the Q4_0 weights and four threads. The baseline is the same GGUF served by a stock library with single-stream decoding. Throughput is in tokens per second.

mode	general	structured	boilerplate
stock GGUF, single-stream	6.6	6.6	6.6
quality (default)	11.2	13.4	16.0
fast	12.5	13.9	17.2
turbo	12.3	13.4	20.4

In quality mode every token is a true argmax of the full model, so the output is the model's own. fast and turbo trade a measure of fidelity for speed on prose and on repetitive output.

Run it

Raspberry Pi OS, 64-bit:

bash

git clone https://github.com/mkturkcan/mote && cd mote
bash scripts/install.sh        # builds the tuned server, fetches the model, installs the gemma command
gemma start                    # quality mode; also: gemma start fast | turbo
gemma chat                     # or open http://<pi-ip>:8080 in a browser
gemma bench                    # measure throughput

The server is OpenAI-compatible on port 8080. The scripts in this repository are the ones used to build, deploy, benchmark, and package the release.

How it works

Decode on a CPU is bound by memory bandwidth: each step streams the entire weight set through RAM, and that read dominates the time. Everything here either amortizes that read or removes a stall the stock engine leaves on the A76.

Speculative decoding. Gemma-4-E2B ships a multi-token-prediction head that drafts tokens ahead. The main model verifies the whole draft in one batched pass and keeps the tokens that match its own greedy output. The A76's int8 matmul reads the weights once for each group of four output rows, so the draft is set three tokens deep: the verify batch is then four wide, three drafts plus the committed token, and lands on exactly one four-row tile. One pass over the weights returns four positions. A fourth draft would widen the batch to five and spill a row into a second pass over the whole matrix, which costs more than the token it buys.

Cortex-A76 kernels. The A76 has no bf16 dot-product instruction, so the stock engine runs the model's bf16 projection weights through a scalar loop. A hand-written NEON kernel, `ggml_vec_dot_bf16`, widens bf16 to f32 in register and accumulates on the NEON pipeline. A second kernel, `ggml_compute_forward_pad_f32`, rewrites the per-step pad over the 256K-token vocabulary, once a per-element bounds-checked copy, as a row-wise memcpy parallelized across the vocabulary.

Arm KleidiAI. KleidiAI's int8 dot-product microkernels, tuned for cores like the A76, take over the Q4_0 weight matmuls that make up the bulk of every forward pass and outrun the engine's own repacked kernel.

Link-time optimization. The engine is compiled whole-program, so the dequantization and dot-product inner loops inline into their callers and the compiler schedules them for the A76 pipeline with the whole program in view.

The model

Mote runs Google's Gemma-4-E2B in Q4_0 GGUF form, with the model's multi-token-prediction head as the speculative drafter.

Gemma-4-E2B carries 2.3B effective parameters and 5.1B in total. Most of the difference is a per-layer embedding table that each decoder layer reads as a lookup rather than multiplying through, so it streams in from storage and the active compute stays at a 2B-class footprint. The text-only GGUF here reports about 4.65B parameters on load, because it keeps that embedding table but drops the vision and audio encoders.

Both files live at unsloth/gemma-4-E2B-it-GGUF, and the installer fetches them. Use of the model is governed by the Gemma Terms of Use. The Mote code in this repository is released under the Apache 2.0 license.

Citation

bibtex

@software{mote_gemma_pi5_2026,
  title  = {Mote: Gemma-4-E2B on a Raspberry Pi 5},
  author = {Mehmet Kerem Turkcan},
  year   = {2026},
  url    = {https://github.com/mkturkcan/mote},
  doi    = {<DOI, generated on Hugging Face>}
}