happyhappy-jun/qwen3-vl-4b-uvd-racer-hl6k-cs16-final

Qwen3-VL-4B — UVD-RACER high-level (HL) subtask planner, RoboCasa365 target CS16 (final labels)

Supervised fine-tune of Qwen/Qwen3-VL-4B-Instruct used as the high-level (HL) stage of the UVD-RACER RoboCasa365 hierarchy: it looks at one tiled camera observation plus the full-task instruction and emits the subtask sentence that conditions the separately published low-level GR00T N1.6 policy. This arm was trained on the 16 RoboCasa365 composite-seen (CS16) target tasks with the final UVD-RACER target subtask annotations. It is not an official Qwen release.

Supersedes the provisional-label HL

This checkpoint supersedes `happyhappy-jun/qwen3-vl-4b-uvd-racer-hl6k`, which is the same recipe, base model and step count trained on the provisional UVD target16 subtask labels (gpu26 job 13433). Only the subtask-label source changed. Prefer this repository (and its cu16 counterpart below) for anything new.

Checkpoint identity

• —Training run: experiment hl_cs in 20260913_target_sub
• —Checkpoint: 20260913_target_sub/hl_cs/checkpoint-6000 (6,000 optimizer updates)
• —Source job: 14078 on the lab AWS cluster gpu26 (us-west-2), 4 x H200 on a single node (partition h200, QoS normal)
• —Completion: COMPLETED, ExitCode 0:0, elapsed 03:15:09, ended 2026-09-13T14:38:52Z (23:38 KST)
• —Base model: Qwen/Qwen3-VL-4B-Instruct at revision ebb281ec70b05090aa6165b016eac8ec08e71b17 (license apache-2.0)
• —Source promotion-manifest SHA-256: 949c6231d429f0b36d13f922a78412b4261196b67093ed777054916e35563896

The run's own COMPLETE.json names checkpoint-6000 ({"global_step": 6000, "microbatch": 32, "effective_batch": 128, "zero_stage": 2, "source_job": "14078"}), trainer_state.json has global_step 6000 == max_steps 6000, and best_model_checkpoint is null (no load-best-at-end) — so checkpoint-6000 is the final model by construction. The published files are copied byte-for-byte from that step. No weights, tokenizer entries or configuration values were changed for this upload; see publication_manifest.json for per-file sizes, SHA-256 values and origins.

Related repositories

What it consumes and what it emits

Input — exactly one image, no frame history. A single 768 x 256 x 3 uint8 RGB tile, the observation.images.robot0_3view_concat stream: three 256 px-wide columns, upright and unscaled, in this order —

[ agentview_left | agentview_right | eye_in_hand ]

Prompt. The user turn is rendered from this template verbatim (template SHA-256 02507401ec656857916b1b825605a0f51bb2ce7ce71a11ef8c18a156af4e74d1, exactly one <image> placeholder):

Task: {task_instruction}

You are observing the robot's current camera view (three views horizontally tiled: left, right, and wrist-mounted).

<image>

What subtask should the robot execute right now? Reply with just the subtask description.

At training time {task_instruction} is the episode's full-task instruction (the dataset's tasks[0]), and the <image> placeholder is expanded in place into the tile's vision tokens. Serve it the same way: one user turn, one image at that position.

Output. One bare subtask sentence, decoded greedily (do_sample=False, num_beams=1, max_new_tokens=64). No chain of thought, no formatting, no list.

Image resolution — the one non-obvious trained input

The training recipe passes --min_pixels 50176 --max_pixels 115200, but under the pinned Transformers 5.6.1 those two arguments are inert: the image processor stores its bounds in a size dataclass, so the fork's update_processor_pixels mutates nothing. The saved processor therefore carries the base bounds {"longest_edge": 16777216, "shortest_edge": 65536}, and the 768 x 256 tile is fed unresized: image_grid_thw [1, 16, 48] = 768 patches = 192 image tokens after the 2 x 2 merge.

This is measured, not inferred, and gated in the eval harness — see experiments/eval_manager/HL_INPUT_CONTRACT.md section 4 in the publishing lab's repository. The processor_config.json shipped here (SHA-256 479b624a5b02852c3947d0b88e2570bb10464d583ad45092e2028099614bd820) is exactly that base-bounds file, so loading the processor from this repository reproduces the trained resolution.

Do not substitute a processor config carrying the recipe's 115200 / 50176 bounds: it shrinks the tile to 576 x 192, image_grid_thw [1, 12, 36], 108 image tokens — a resolution this model never saw.

Training setup

Supervision and frame sampling

• —Targets are the per-frame subtask sentences of the final UVD-RACER target annotation delivery (derived segment key subtask_uvd_final, sentence column annotation.human.subtask).
• —Frames are sampled every_k: train stride 2, offset 0; val stride 40, offset 1; an edge buffer of 3 frames is dropped on each side of a segment boundary; tiles are encoded as JPEG quality 90.
• —The HL sees no action, state or proprioception input — only the tile and the instruction text.

Contents

model.safetensors (single unsharded bf16 weight file), config.json, generation_config.json, tokenizer.json, tokenizer_config.json, chat_template.jinja, processor_config.json, plus this card and publication_manifest.json.

This repository is self-sufficient for inference — model, tokenizer and processor all load from it directly:

from transformers import AutoProcessor, AutoModelForImageTextToText

REPO = "happyhappy-jun/qwen3-vl-4b-uvd-racer-hl6k-cs16-final"
processor = AutoProcessor.from_pretrained(REPO)
model = AutoModelForImageTextToText.from_pretrained(REPO, dtype="auto", device_map="auto")
# greedy decoding, as trained and evaluated:
#   model.generate(**inputs, do_sample=False, num_beams=1, max_new_tokens=64)

vocab.json / merges.txt are not shipped and are not needed: tokenizer.json is self-contained, so the fast tokenizer loads without them.

Where each file comes from

The run wrote two exports at step 6,000: the checkpoint directory 20260913_target_sub/hl_cs/checkpoint-6000, which the run's own completion marker designates, and a final save_model export at the run root. The six weight/config/tokenizer files here are copied byte-for-byte from `checkpoint-6000`. The pinned trainer passes only the tokenizer to the Trainer as its processing class, so no checkpoint-N/ directory contains a processor config; `processor_config.json` is therefore copied byte-for-byte from the run-root `save_model` export — the same step of the same job — purely so that AutoProcessor.from_pretrained resolves against this repository. It is a Qwen3VLProcessor config holding the Qwen2VLImageProcessor settings (patch size 16, merge size 2, temporal patch size 2, mean/std 0.5) and the video-processor settings (fps 2, frame sampling on); on this Transformers version that single file replaces the base repo's separate preprocessor_config.json and video_preprocessor_config.json. No preprocessing setting was changed relative to the base model. Per-file sizes and SHA-256 values, each tagged with its origin, are in publication_manifest.json.

Not included, by design: DeepSpeed ZeRO optimizer shards (global_step6000/), optimizer/scheduler/RNG state, trainer_state.json, training_args.bin, training logs and the training data. This is a model export for inference, not a complete optimizer-state resume bundle.

Download

from huggingface_hub import snapshot_download

checkpoint_dir = snapshot_download(
    "happyhappy-jun/qwen3-vl-4b-uvd-racer-hl6k-cs16-final",
    # revision=<commit sha of the single publication commit>
)

The whole payload is submitted in one atomic commit on a freshly created repository, so that commit's sha is the only model revision here. It cannot be written inside this file (the text would change the commit it names); the pinned snapshot_download(..., revision=...) snippet is recorded in the publishing lab's publication log alongside the per-file SHA-256 values.

Limitations

No GPU inference or robot-rollout evaluation was run for this upload. Upload verification covers source checksum identity, the Safetensors header and contiguous tensor-offset coverage of the single weight file, payload content scanning, and Hub file integrity after the commit — nothing more. No subtask-accuracy, task-success, hierarchical-policy or safety-performance number is claimed by this model card, and none should be inferred from the training losses above: they are training/validation cross-entropy on the sampled frames of this split, not a behavioural metric.

Further scope limits worth stating plainly: the model was trained on 16 target tasks of one simulated benchmark with a fixed three-view tile, emits language only (it drives no actuator by itself), and was optimized for composite-seen (CS16) RoboCasa365 tasks — behaviour on other task distributions, camera layouts or tile orders is untested.

License and safety

Inherits the base model's Apache-2.0 license. Attribution: Qwen team, Alibaba Cloud — Qwen/Qwen3-VL-4B-Instruct. This repository is an independent derivative fine-tune and is not endorsed by the Qwen team.

Outputs are generated language and are not guaranteed to be correct or safe. Validate in simulation and use appropriate robot safety controls before any physical deployment.