DorianAtSchool/qwen3vl-8b-robocasa-v3-active-observation-causal

qwen3vl-8b-robocasa-v3-active-observation-causal

LoRA adapter fine-tuning Qwen/Qwen3-VL-8B-Instruct for RoboCasa tool-calling agent behavior with causal-consistent active-observation caching.

Model Description

This is a follow-up to `qwen3vl-8b-robocasa-v3-active-observation` that fixes a training/eval consistency bug in how image observations were attached to training examples.

The bug: the original v3 training pipeline decided whether to attach the latest observation image to an example based on last_agent — effectively conditioning on which agent had most recently produced the acting example, which in practice let the model infer the correct next tool call from whether an image was present at all, rather than from genuine visual/task reasoning. This "cheat code" (image present → call a comm/physical tool; image absent → call get_image) is not available at inference time in the same closed-loop way it appeared during training, so it broke causal consistency between train and eval.

The fix: introduced a causal_single_cache mode that tracks an explicit active_observation_agent cache during example construction. The cached image owner only updates when a get_image call occurs, and resets to None whenever a different agent takes a non-communicate/non-task-complete action. Image attachment now depends only on this causally-prior state, not on which tool call comes next in the trajectory.

Training Details

Training Data

Same RoboCasa AgentSFT subset as the base v3 run — 46 kitchen tasks (see run_config.json for the full task list), tool-call SFT format.

Training Hyperparameters

• —LoRA rank (r): 16
• —LoRA alpha: 32
• —LoRA dropout: 0.05
• —Target modules: o_proj, q_proj, k_proj, v_proj, down_proj, gate_proj, up_proj
• —Task type: CAUSAL_LM
• —Attention implementation: FlashAttention 2
• —Image resolution: 512
• —Max sequence length: 8192
• —Per-device batch size: 8
• —Gradient accumulation: 1
• —Epochs: 3
• —Flags: --predict-acting-agent --train-get-image --causal-single-cache

Hardware

8x NVIDIA B200 (single node), ~2h30m wall clock.

Results

W&B run: qwen3vl-8b-v3-active-observation-causal

Usage

from peft import PeftModel
from transformers import AutoModelForVision2Seq, AutoProcessor

base_model = AutoModelForVision2Seq.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")
model = PeftModel.from_pretrained(base_model, "DorianAtSchool/qwen3vl-8b-robocasa-v3-active-observation-causal")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")

Framework versions

• —PEFT 0.19.1