DorianAtSchool/RoboTalk-Qwen3-VL-8B-Instruct-150traj

RoboTalk-Qwen3-VL-8B-Instruct-150traj

One-epoch LoRA adapter trained on 150 RoboTalk trajectories per training task across 43 tasks (6,450 training trajectories). Ten additional tasks are held out. The prediction target is the next tool call. One shared policy controls two agents with separate partially observable contexts and a coordinator–follower protocol.

Load the adapter

This repository is an adapter, not a standalone model. It requires the matching Qwen model and processor. Use Transformers with Qwen3-VL support and PEFT.

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel

base_id = "Qwen/Qwen3-VL-8B-Instruct"
processor = AutoProcessor.from_pretrained(base_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    base_id, torch_dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(model, "DorianAtSchool/RoboTalk-Qwen3-VL-8B-Instruct-150traj")
model.eval()

Use the model's chat template for tool calling. Reproducing the scores requires the RoboTalk task goal, initial state, global tool definitions, per-agent visual observations and private histories, and full communication protocol—not a generic chat prompt. Histories do not include step indices. See the dataset for trajectories and prompt metadata.

Training

• —Base model: Qwen/Qwen3-VL-8B-Instruct.
• —One epoch, learning rate 1e-4, weight decay 0.01, cosine schedule, 3% warmup.
• —LoRA rank 16, alpha 32, dropout 0.05.
• —Effective batch size 64 across four GPUs; sequence limit 8,192 tokens including visual tokens.
• —Seed 42; the released adapter contains language-model LoRA weights, not a separately fine-tuned vision backbone.
• —Larger scale datasets contain the smaller-scale selections.

Full sanitized settings are in training_config.json; task membership is in task_split.json. Optimizer states and training-machine paths are not included.

Closed-loop evaluation

Full communication, ten episodes per task, native 43/10 fixed-cohort sampling. Error-free FSM success means reaching the symbolic goal with no rejected calls. The intervals are 95% Wilson intervals over episodes, not over newly sampled tasks.

Exact counts and evaluation settings are in evaluation_results.json. These results use the actual one-epoch checkpoint and corrected cohort selections. Instruct evaluation uses temperature 0 and a 256-token output budget; Thinking evaluation uses temperature 0.6, top-p 0.95, top-k 20 and a 2,048-token output budget. These inference settings differ between model variants.

Scope and limitations

Intended for research on high-level coordination in a household simulator, not direct hardware control or safety-critical deployment. FSM success is a symbolic task metric and need not imply satisfaction of every native physical criterion. The underlying Qwen model is distributed under Apache-2.0; consult its model repository for its terms. This card does not assign a separate adapter license.