CoolFace
Modelpublic

GatekeeperZA/Qwen3-4B-Instruct-2507-RKLLM-v1.2.3

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face
0likes181downloads
Model Card

Qwen3-4B-Instruct-2507 — RKLLM v1.2.3 (w8a8, RK3588)

RKLLM conversion of Qwen/Qwen3-4B-Instruct-2507 for Rockchip RK3588 NPU inference.

Converted with RKLLM Toolkit v1.2.3. The 2507 suffix denotes the July 2025 refresh of Qwen3-4B with improved instruction following.

Note: This model does not produce <think> reasoning blocks in this RKLLM build (thinking mode is disabled). See RKLLM thinking mode limitations.

Key Details

PropertyValue
Base ModelQwen/Qwen3-4B-Instruct-2507
Toolkit VersionRKLLM Toolkit v1.2.3
Runtime VersionRKLLM Runtime ≥ v1.2.1 (v1.2.3 recommended)
Quantizationw8a8, group size 128
Target PlatformRK3588
NPU Cores3
Max Context Length16384 tokens
Optimization Level1
Hybrid Ratio0.0
Thinking Mode❌ Disabled
LanguagesEnglish, Chinese (multilingual)

Why This Model?

Qwen3-4B-Instruct-2507 is the July 2025 update to Alibaba's Qwen3-4B, with improved reasoning and instruction following. At 4B parameters it is the largest text-only model in this RK3588 lineup and handles complex prompts well despite the quantization.

The 16k context window (vs 8k on smaller models) enables long document summarisation and multi-turn conversations.

Hardware Tested

  • —Orange Pi 5 Plus — RK3588, 16GB RAM, Armbian Linux
  • —RKNPU driver 0.9.8
  • —RKLLM Runtime v1.2.3

Usage

With the RKLLM API Server

bash
mkdir -p ~/models/Qwen3-4B-Instruct-2507
cd ~/models/Qwen3-4B-Instruct-2507
git lfs install && git clone https://huggingface.co/GatekeeperZA/Qwen3-4B-Instruct-2507-RKLLM-v1.2.3 .

Use with GatekeeperZA/RKLLM-API-Server.

With the official RKLLM API demo

bash
git clone https://github.com/airockchip/rknn-llm.git
cd rknn-llm/examples/rkllm_api_demo
./build/rkllm_api_demo /path/to/Qwen3-4B-Instruct-2507-rk3588-w8a8_g128-opt-1-hybrid-ratio-0.0-16k.rkllm 8192 16384

File Listing

FileDescription
Qwen3-4B-Instruct-2507-rk3588-w8a8_g128-opt-1-hybrid-ratio-0.0-16k.rkllmQuantized model for RK3588 NPU

Compatibility Notes

  • —Minimum runtime: RKLLM Runtime v1.2.1. v1.2.3 recommended.
  • —RKNPU driver: ≥ 0.9.6
  • —SoCs: RK3588 / RK3588S. Not compatible with RK3576 without reconversion.
  • —RAM: ~5.5GB loaded. Requires 8GB+ board (16GB recommended).

Acknowledgements

  • —Alibaba Qwen Team for Qwen3
  • —Rockchip / airockchip for the RKLLM toolkit and runtime
  • —Converted by GatekeeperZA