DorianAtSchool/qwen3vl-8b-robocasa-v3-active-observation

qwen3vl-8b-robocasa-v3-active-observation

LoRA adapter fine-tuning Qwen/Qwen3-VL-8B-Instruct for the RoboCasa behavior-cloning task-VLM ("v3 active-observation") pipeline: the model predicts the acting agent and is trained with --train-get-image, i.e. it also learns to actively request observation images during rollout.

Training details

• —Base model: Qwen/Qwen3-VL-8B-Instruct
• —Method: LoRA SFT via PEFT, rank 16, alpha 32, dropout 0.05
• —target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —Attention: FlashAttention-2, image resolution 512, max sequence length 8192
• —Hardware: 8x NVIDIA B200, single node, accelerate MULTI_GPU (DDP)
• —Schedule: 3 epochs, per-device batch size 8, gradient accumulation 1
• —Flags: --predict-acting-agent --train-get-image

Results

Logged to Weights & Biases: robocasa-bc-task-vlm/l923azcv

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-8B-Instruct", torch_dtype="bfloat16", device_map="auto",
    attn_implementation="flash_attention_2",
)
model = PeftModel.from_pretrained(base, "DorianAtSchool/qwen3vl-8b-robocasa-v3-active-observation")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

Framework versions

• —PEFT 0.19.1
• —transformers 5.13.0