jucamohedano/qwen3-vl-4b-oven-grpo-aggregation-prelim

Qwen3-VL-4B OVEN GRPO — aggregation prelim (LoRA)

LoRA adapter for `Qwen/Qwen3-VL-4B-Instruct`, trained with GRPO on the OVEN open-domain visual entity recognition task using verl.

Preliminary (pre-n16) aggregation run; superseded by the n16 checkpoint.

Exploratory run from the training sweep, not reported individually in the thesis. Released to document the search over reward designs, prompt formats, and data curations.

Recipe

• —Base model: Qwen/Qwen3-VL-4B-Instruct
• —Adapter: LoRA, rank 32, alpha 32 (task_type: CAUSAL_LM)
• —Algorithm: GRPO (verl), group size 8, seed 42, ~299 steps
• —Prompt / reward: 80% aggregation prompts, one-shot. Shaped OVEN reward (format 0.05, exact/fuzzy 0.70, specificity-weighted hF 0.15, path-match 0.05, aggregation 0.05); a missing \boxed{} answer returns 0.00.

Training dynamics (from the wandb run)

Training reward moved 0.158 → 0.145 while held-out exact-match went 0.139 → 0.139 (essentially flat). Consistent with the thesis: outcome-only GRPO does not raise validation accuracy above the prompt-only elicitation floor. As a small-KL LoRA update the policy stays close to the reference model.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3-VL-4B-Instruct", trust_remote_code=True, device_map="auto")
model = PeftModel.from_pretrained(base, "jucamohedano/qwen3-vl-4b-oven-grpo-aggregation-prelim")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct", trust_remote_code=True)

Intended use

Research artifact for reproducing the thesis's (largely negative) reinforcement-learning results on OVEN and for studying GRPO training dynamics on a multimodal open-world task. Not tuned or recommended for production classification.

Provenance

MSc thesis, University of Trento (Juan Camacho Mohedano). Training: verl (branch grpo-oven-v080); reward in verl/utils/reward_score/oven_boxed.py; data by oven-mllm-eval/scripts/build_verl_oven_parquet.py. wandb run: offline-run-20260707_001121-qwen3-vl-4b-oven-grpo-agg08-one-shot-3k-shaped-seed42.