OpenRAL/rskill-qwen35_4b-any-general-nf4

rskill-qwen35_4b-any-general-nf4

OpenRAL rSkill — Qwen3.5-4B natively-multimodal video-language model packaged as an NF4 bitsandbytes vlm rSkill. Accepts RGB image or video frames plus a natural-language query; returns a text answer. No actuators. Apache-2.0.

Quick Start

ral skill install hf://OpenRAL/rskill-qwen35_4b-any-general-nf4

from openral_core.schemas import RSkillManifest

manifest = RSkillManifest.from_yaml("rskills/qwen35-4b-nf4/rskill.yaml")
assert manifest.kind == "vlm"
assert manifest.role == "s2"
assert manifest.quantization.extra["scheme"] == "nf4"
assert manifest.is_commercial_use_allowed is True

What It Does

Qwen3.5-4B is a natively-multimodal foundation model trained from scratch on interleaved text, image, and video tokens. Given an RGB image or video clip and a natural-language question, it returns a free-form text answer grounded in the visual content.

This rSkill declares kind: vlm and role: s2 because it is a pure perception component operating at S2 (slow-reasoning) rate (~0.2–1 Hz), not an S1 fast policy. It consumes camera frames and natural-language queries, emits text answers, and never drives `ros2_control` joints.

Representative queries for robot scene understanding:

• —"What objects are on the table?"
• —"Is the gripper clear of obstacles?"
• —"Describe the relative positions of the cup and bowl."
• —"Has the pick-and-place task completed?"

Why Qwen3.5-4B over Qwen2.5-VL-7B

Qwen3.5-4B beats Qwen2.5-VL-7B on every video benchmark despite being 3B smaller. The 3:1 Gated DeltaNet / full-attention hybrid processes long video sequences far more efficiently — important for continuous robot camera streams. At NF4 it fits well within 8 GB VRAM alongside the S1 skill stack.

Architecture

Qwen3.5 uses a 3:1 hybrid attention stack: three Gated DeltaNet (linear-attention, O(n)) layers for every one full-attention layer. This reduces cost on long sequences significantly. The vision encoder is shared with Qwen3-VL. Key features:

• —Native video support — temporal patch embedding, second-level event localization, up to 256K context (extensible to 1M)
• —Spatial grounding — RefCOCO avg ~80.6; strong for "where is X?" queries
• —201-language support

Runtime

This rSkill ships a pre-quantized NF4 checkpoint as weights_uri (hf://OpenRAL/rskill-qwen35_4b-any-general-nf4): model.safetensors with an embedded bitsandbytes quantization_config (nf4, double-quant, bf16 compute). The sidecar loads it directly as 4-bit (~3.3 GB resident, no bf16 load spike), so it fits an 8 GB GPU with no loader workaround. source_repo records the SHA-pinned upstream Apache-2.0 model it was quantized from (provenance, §8).

Reproduce the checkpoint with tools/build_qwen_vlm_nf4_checkpoint.py (run in the sidecar venv):

$OPENRAL_QWEN_VLM_SIDECAR_VENV/bin/python tools/build_qwen_vlm_nf4_checkpoint.py \
  --source Qwen/Qwen3.5-4B \
  --out ~/.cache/openral/qwen35-4b-nf4-ckpt

It loads the upstream model once (forcing serial materialization so the bf16 pass fits 8 GB), saves the NF4 weights + processor, then verifies the checkpoint reloads directly as 4-bit and answers a smoke query.

The kind: vlm runtime is implemented as a read-only reasoner tool, not an ExecuteSkill (a scene VLM produces text, not actions):

• —Sidecar: tools/qwen_vlm_sidecar.py boots the NF4 model in its own venv and serves a ZMQ REQ/REP + msgpack protocol. Provision it separately and point at it with OPENRAL_QWEN_VLM_SIDECAR_VENV (or let the backend auto-spawn it on first query).
• —Backend: openral_runner.backends.gstreamer.qwen_scene_vlm.QwenSceneVlm is the node-side ZMQ client; build_scene_vlm(manifest) builds it from this manifest. The node-side client deps (pyzmq + msgpack) install with uv sync --group qwen-vlm.
• —Service node: openral_perception_ros.scene_vlm_node subscribes the cameras and serves /openral/perception/query_scene (openral_msgs/srv/QueryScene).
• —Reasoner tool: the LLM sees the read-only query_scene tool when the reasoner is launched with scene_query_available:=true. It asks open-ended scene-state questions ("has the robot grasped the mug?", "is the task complete?") and the answer feeds the next reasoning tick.

Validated live

The sidecar + backend + query_scene path was run end-to-end on an NVIDIA RTX 4070 Laptop (8 GB): NF4 Qwen3.5-4B loads to ~3.3 GB resident, and real image queries return correct answers — including the task-verification use case ("Has a robot gripper grasped any object?" → "No", grounded in the frame). Covered by the GPU-gated tests/unit/test_qwen_scene_vlm.py::test_e2e_query_coco_sample (set OPENRAL_QWEN_VLM_SIDECAR_VENV), not asserted blind.

8 GB load note. Deploying the pre-quantized weights_uri loads the 4-bit weights directly (~3.3 GB, ~6 s) with no workaround — the clean 8 GB path. The workaround only matters when quantizing at load from the raw upstream (the build step, or --model Qwen/Qwen3.5-4B): transformers 5.x's parallel loader materializes weights in bf16 on-GPU before bitsandbytes quantizes, and the 4-way-concurrent ~7.4 GB transient OOMs an 8 GB card, so the sidecar forces serial materialization (core_model_loading.GLOBAL_WORKERS = 1) + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. The sidecar auto-detects which path applies. The Gated-DeltaNet fast kernels (fla / causal-conv1d) are optional — without them transformers uses a slower torch fallback (the model still loads and answers). The model loads via AutoModelForImageTextToText (it registers as Qwen3_5ForConditionalGeneration).

Benchmark Numbers

Benchmarks below are paper-reported (Qwen team, February 2026); reproduced_locally: false in the eval JSON.

Supported robots and embodiments

This scene VLM is embodiment-agnostic — it reasons about camera frames and emits text, never actuator commands, so it imposes no kinematic requirement. The only hardware dependency is an RGB camera stream of at least 336×336. All in-tree OpenRAL embodiment tags are therefore listed in rskill.yaml (aloha, franka_panda, g1, google_robot, gr1, h1, mobile_base, openarm, panda_mobile, pusht, rizon4, sawyer, so100_follower, so101_follower, ur10e, ur5e, widowx) so any robot with a compatible camera can install it and expose the reasoner's query_scene tool. It pairs with any S1 VLA policy: the VLA acts, this VLM verifies (e.g. "did the grasp succeed?").

Sensors and Observation Contract

The model emits no action chunks and has no proprioception contract.

Manifest Summary

License

The rSkill package metadata and README are OpenRAL project files under Apache-2.0. The wrapped Qwen3.5 weights are released by the Qwen Team under Apache-2.0, permitting commercial use. No OPENRAL_ALLOW_NONCOMMERCIAL=1 flag is needed.