Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-64k-balanced

Qwen3-VL-8B-Instruct-UI-Genie-scoring-64k-balanced

A Bradley-Terry reward model fine-tuned from Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie (itself SFT'd from Qwen/Qwen3-VL-8B-Instruct) on a class-balanced 64k subset of UI-Genie-RM-517k.

This is the balanced-split counterpart to Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring. Same architecture, same objective, different training subset.

Merged sequence-classification checkpoint (task_type="seq_cls", num_labels=1, problem_type="regression") trained with the Bradley-Terry pairwise loss. A learned score.weight linear head projects the last non-padding token's hidden state to a scalar reward.

Score = `score_head(last non-padding token hidden state)` — a single float per input trajectory. Higher score = higher quality action.

Provenance of the head is recorded in score_head_source.txt: consolidated from the deepspeed shards of bt_qwen3vl_8b_64k_balanced/checkpoint-1000/global_step*/ via zero_to_fp32. The swift trainer keeps the score head out of the LoRA adapter and writes it only to deepspeed checkpoint shards, so a naive merge/export silently drops it — this checkpoint carries the actual trained head.

Intended Use

Scalar reward signal for GUI agent training (PPO/GRPO, RLHF) or best-of-N selection. Unlike the discrete <|+|> / <|-|> classifier, this model outputs a continuous unbounded reward directly usable as a value signal.

Evaluation

Pairwise accuracy on android_flux_recovery_action_preference from Gyubeum/AndroidFlux_RM_Eval — 94 pairs, each two candidate t+1 recovery actions from the same checkpoint, labelled by success under a 12-model continuation panel:

This checkpoint performs at chance on that benchmark. Breaking it out by subset shows no split above chance:

For comparison, the unbalanced -scoring model separates the same pairs with visible structure (0.658 on error_path vs 0.571 on clean_path). The head here does produce well-separated scores — only 1 tie, margins spanning [−17.75, +18.38] — but that separation does not correlate with which action succeeds.

With n=94 the 95% CI is roughly ±10 points, so treat the gap as indicative rather than tight. Evaluated 2026-08-27 with rm_eval --mode bt.

Inference

vLLM's Qwen3VLForConditionalGeneration loader does not support the extra score.weight tensor — use HuggingFace transformers directly.

import json, torch
from safetensors import safe_open
from huggingface_hub import hf_hub_download
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

MODEL_PATH = "Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-64k-balanced"
DEVICE = "cuda"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL_PATH, torch_dtype=torch.bfloat16, low_cpu_mem_usage=True,
).to(DEVICE).eval()
processor = AutoProcessor.from_pretrained(MODEL_PATH, max_pixels=1_048_576)

# `score.weight` is not wired into the base CausalLM class — load it separately.
index = json.load(open(hf_hub_download(MODEL_PATH, "model.safetensors.index.json")))
shard = hf_hub_download(MODEL_PATH, index["weight_map"]["score.weight"])
with safe_open(shard, framework="pt") as f:
    w = f.get_tensor("score.weight")
score_head = torch.nn.Linear(w.shape[1], w.shape[0], bias=False)
with torch.no_grad():
    score_head.weight.copy_(w)
score_head = score_head.to(DEVICE, dtype=torch.bfloat16)

# score = score_head(hidden_states[:, last_non_pad_index, :])

Training Details

Related Models

• —[Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring](https://huggingface.co/Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring) — same objective on the unbalanced 64k split; higher accuracy on the recovery benchmark.
• —[Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie](https://huggingface.co/Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie) — the SFT discrete preference classifier this model's score.weight was fine-tuned on top of.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
}