Mojo24x7/gemma-4-E2B-it-rkllm-rk3588

gemma-4-E2B-it — RKLLM w8a8 for Rockchip RK3588

google/gemma-4-E2B-it converted to Rockchip's RKLLM format so it runs on the RK3588 NPU. Built and verified on a Radxa ROCK 5B+ (16 GB).

At the time of conversion there was no gemma-4 RKLLM build published anywhere, so this fills a real gap: the E2B is the gemma-4 variant that is actually deployable on RK3588 — see the E4B does not load section below, which is the other half of the finding.

Requirements

• —RK3588 / RK3588S board with ≥ 16 GB RAM (the model alone is 7.3 GiB resident)
• —rknpu kernel driver — check with sudo cat /sys/kernel/debug/rknpu/version (expect v0.9.8)
• —RKLLM runtime 1.3.0 (librkllmrt.so) from airockchip/rknn-llm

The NPU driver is not optional and cannot be worked around — with no /sys/kernel/debug/rknpu there is no backend to run on, regardless of anything else.

★ Required: set the chat template yourself

gemma-4's embedded chat template is not parseable by the RKLLM runtime. You will see:

W rkllm: Failed to parse chat_template: invalid format detected

and then the model emits one token and stops — an empty reply. It is not a bad conversion; the runtime simply applies no framing, so the model never sees a turn structure. Call rkllm_set_chat_template() explicitly:

With that set, output is clean, coherent English. Without it you get nothing, which is a confusing failure mode — hence this being the first thing in the card.

Measured

On a ROCK 5B+ (RK3588, 16 GB, performance governors, 3 NPU cores):

• —model load: 4.1 s
• —no rkllm_init errors, no allocation failures
• —coherent English, correct factual recall, no language drift
• —sampling: the runtime's built-in gemma profile is appropriate (top_k 40 / top_p 0.95 / temperature 0.7 / repeat_penalty 1.1)

Throughput was not benchmarked in a controlled way, so no tokens/s figure is quoted here rather than publishing a number that was not properly measured.

★ The E4B sibling does NOT load on RK3588 (rknpu 0.9.8)

gemma-4-E4B-it converts successfully with the same toolchain, but cannot be loaded by the RKLLM runtime on rknpu driver 0.9.8:

E RKNN: failed to allocate handle, errno 14 Bad address;
        failed to malloc npu memory, size: 117637120 (117 MB), flags 0x2
E rkllm: rkllm_init failed

This was chased to a conclusion and it is not a memory-configuration problem:

• —fails identically at CMA 256 MB and CMA 1 GB
• —fails on a freshly booted, otherwise idle board with 14.6 GB RAM free and 970 MB of CMA free — i.e. plenty of contiguous memory available
• —CmaFree stays flat during the failing allocation, so nothing is being exhausted
• —fails identically on two different RK3588 boards
• —a Qwen3.5-9B model of larger total size loads fine on the same driver

The 117 MB request is a single NPU buffer sized by gemma-4's 262 144-token vocab. The driver rejects that single allocation outright (EFAULT), regardless of free memory. A smaller-vocab model of greater total size is unaffected. So it is a driver single-allocation limit, and E2B is the gemma-4 that works on this stack today.

If you have a newer rknpu driver, E4B may be worth retrying — the conversion side is fine.

Reproducing the conversion

Included: convert_text_rkllm.py and data_quant_gemma.json (the calibration set).

# on an x86_64 host — rkllm-toolkit has no aarch64 wheel, so the boards cannot convert
pip install rkllm-toolkit==1.3.0
python convert_text_rkllm.py  <path-to>/gemma-4-E2B-it  data_quant_gemma.json

★ Calibration data matters more than people expect. W8A8 calibrates activation ranges against whatever distribution you feed it, and getting this wrong produces models that look fine on trivial prompts and fall apart on real ones. Two failures worth knowing about, both hit on this hardware:

1. 1.The calibration file shipped in the RKLLM examples is Chinese. Using it for an English model produced output that drifted into Chinese and Polish and ignored English instructions — while still answering "what is the capital of France" correctly, which is exactly why it is easy to miss.
2. 2.Calibrating a text model on image-text samples (from the multimodal export path) produced degenerate, repetitive output.

So: calibrate on text that matches your target language and task, and always check the output language, not merely that tokens come out. data_quant_gemma.json here is 72 English instruction/response samples framed with gemma's own turn markers (<start_of_turn> / <end_of_turn>), verified to contain no CJK and no empty entries.

Credits

• —[google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) — the base model, Apache-2.0. All model capability is Google's; this repo only changes the numeric format.
• —[airockchip/rknn-llm](https://github.com/airockchip/rknn-llm) — Rockchip's RKLLM toolkit and runtime.
• —Thanks to Angelo Saraceno, whose write-up on running gemma-4 on RK3588 confirmed the E2B text path was viable before we spent conversion time on it.

License

Apache-2.0, inherited from the base model. This is a format conversion of google/gemma-4-E2B-it; no weights were retrained, merged or otherwise modified beyond quantization. Use is subject to the base model's terms.