n8dgr8/ov_intent_analysis_sft-RKLLM-RK3588

1---2license: apache-2.03base_model:4  - guoxuter/ov_intent_analysis_sft5tags:6  - rockchip7  - rk35888  - rkllm9  - npu10  - openviking11  - retrieval12  - intent-analysis13  - query-planning14  - qwen3.515pipeline_tag: text-generation16---17 18# ov_intent_analysis_sft — RKLLM (RK3588) conversion19 20Pre-converted **W8A8** RKLLM runtime file of [`guoxuter/ov_intent_analysis_sft`](https://huggingface.co/guoxuter/ov_intent_analysis_sft) (the OpenViking retrieval intent-analysis / query-planner model, fine-tuned from Qwen3.5-0.8B).21 22Run the **same tuned query planner** on your RK3588 NPU — no x86 conversion rig required.23 24## Why this exists25 26OpenViking's docs recommend this model for the `query_planner` slot: it decides whether a search needs context retrieval, skips chitchat (no queries → no token spend), and emits structured `skill` / `resource` / `memory` queries. The stock model ships as HF safetensors; to run it on the RK3588 NPU you need a `.rkllm` file, which can only be produced by **rkllm-toolkit (x86_64-only)**. This repo is that conversion, done once, so RK3588 owners can skip the whole rig.27 28## File29 30| File | Size | Spec |31|---|---|---|32| `ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm` | 1.3 GB | W8A8, RK3588, 3 NPU cores, max_context 4096 |33 34Runtime requirements: `librkllmrt.so` 1.3.0 (rkllm-toolkit 1.3.0 generation). Verified on kernel 6.1 vendor with rknpu driver 0.9.8.35 36## How to serve it37 38Any RKLLM-capable server works. Two options:39 40### Option A — full rkllama server (Ollama API; recommended for OpenViking)41 42```bash43pip install rkllama   # python 3.9–3.1244rkllama_server --models /path/to/models45```46 47Place the `.rkllm` in your models dir. This gives an Ollama-compatible API, which is what OpenViking's `query_planner` speaks.48 49### Option B — minimal OpenAI+Ollama server (no transformers/torch)50 51The same ctypes wrapper, no heavy deps (see the conversion recipe below for the gist). Serves `/v1/*` and `/api/*`.52 53## OpenViking wiring54 55```json56{57  "query_planner": {58    "provider": "litellm",59    "model": "ollama/guoxuter/ov_intent_analysis_sft:v7_q8",60    "api_base": "http://127.0.0.1:8091",61    "temperature": 0.0,62    "timeout": 60,63    "extra_request_body": { "think": false }64  }65}66```67 68**Keep the model string exactly as-is** — OpenViking auto-matches the bundled v7 prompt by string (`retrieval.ov_intent_analysis_sft_v7` in `intent_analyzer.py`). Only `api_base` changes: point it at your RKLLM server instead of Ollama.69 70## Benchmark (RK3588, same prompt)71 72| Runtime | Wall time | Notes |73|---|---|---|74| Ollama / CPU (GGUF Q8) | 16.5 s | output lands in `thinking` unless `think:false` |75| rk-llama.cpp NPU (GGUF Q8) | 10.9 s | needed `--reasoning off` |76| **RKLLM NPU (this file, W8A8)** | **~9 s** (1.8 s warm) | prefill ~200 t/s, decode ~13 t/s |77 78Query planning is prefill-dominated, which is exactly where the NPU wins. Decode is memory-bandwidth-bound, so don't expect magic on long generations — this model's outputs are short JSON.79 80## Conversion recipe (for reproducing / other models)81 82The converter is **x86-only**, so run it on an x86 box (any Linux, or a serverless cloud like Modal):83 84```python85# rkllm-toolkit 1.3.0 (wheel from airockchip/rknn-llm release-v1.3.0,86# rkllm-toolkit/packages/rkllm_toolkit-1.3.0-cp311-cp311-linux_x86_64.whl)87# deps pinned from that release's requirements.txt (torch 2.6.0, transformers 5.8.0, ...)88 89from rkllm.api import RKLLM90 91llm = RKLLM()92llm.load_huggingface(model="guoxuter/ov_intent_analysis_sft", device="cpu")  # or cuda93llm.build(94    do_quantization=True,95    optimization_level=1,96    quantized_dtype="W8A8",97    quantized_algorithm="normal",98    target_platform="RK3588",99    num_npu_core=3,100    dataset="data_quant.json",   # calibration: input/target pairs101    hybrid_rate=0,               # REQUIRED arg in 1.3.0102    max_context=4096,            # NOTE: `max_context`, NOT `max_context_len`103)104llm.export_rkllm("ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm")105```106 107Gotchas hit along the way (so you don't):108- `build()` takes `max_context`, not `max_context_len` (raises `TypeError: unexpected keyword argument`).109- `hybrid_rate=0` is required in the 1.3.0 signature.110- The toolkit only ships **x86_64 wheels** — there is no aarch64 path; don't fight it on an ARM SBC.111- Keep the toolkit major version aligned with your runtime's `librkllmrt.so` (1.3.0 ↔ 1.3.0).112 113## Attribution & license114 115- Base model: [`guoxuter/ov_intent_analysis_sft`](https://huggingface.co/guoxuter/ov_intent_analysis_sft) — **Apache-2.0**, which its card states covers the fine-tuned checkpoint (same license as Qwen3.5-0.8B).116- Conversion performed with Rockchip's rkllm-toolkit 1.3.0 ([airockchip/rknn-llm](https://github.com/airockchip/rknn-llm)).117- This conversion is published under **Apache-2.0**.118 119Big thanks to guoxuter for the tuned model and the OpenViking team for the recommended workflow.120