jucamohedano/qwen3-vl-4b-oven-grpo-traversal-wikidata-shaped

Qwen3-VL-4B OVEN GRPO — traversal wikidata shaped (LoRA)

LoRA adapter for `Qwen/Qwen3-VL-4B-Instruct`, trained with GRPO on the OVEN open-domain visual entity recognition task using verl.

Wikidata-format traversal with the shaped reward.

Exploratory run from the training sweep, not reported individually in the thesis. Released to document the search over reward designs, prompt formats, and data curations.

Recipe

• —Base model: Qwen/Qwen3-VL-4B-Instruct
• —Adapter: LoRA, rank 64, alpha 32 (task_type: CAUSAL_LM)
• —Algorithm: GRPO (verl), group size 16, seed 42, ~300 steps
• —Prompt / reward: traversal, 'wikidata' format, shaped reward. Shaped OVEN reward (format 0.05, exact/fuzzy 0.70, specificity-weighted hF 0.15, path-match 0.05, aggregation 0.05); a missing \boxed{} answer returns 0.00.

Training dynamics (from the wandb run)

Training reward moved 0.102 → 0.072 while held-out exact-match went 0.144 → 0.142 (essentially flat). Consistent with the thesis: outcome-only GRPO does not raise validation accuracy above the prompt-only elicitation floor. As a small-KL LoRA update the policy stays close to the reference model.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained("Qwen/Qwen3-VL-4B-Instruct", trust_remote_code=True, device_map="auto")
model = PeftModel.from_pretrained(base, "jucamohedano/qwen3-vl-4b-oven-grpo-traversal-wikidata-shaped")
processor = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-4B-Instruct", trust_remote_code=True)

Intended use

Research artifact for reproducing the thesis's (largely negative) reinforcement-learning results on OVEN and for studying GRPO training dynamics on a multimodal open-world task. Not tuned or recommended for production classification.

Provenance

MSc thesis, University of Trento (Juan Camacho Mohedano). Training: verl (branch grpo-oven-v080); reward in verl/utils/reward_score/oven_boxed.py; data by oven-mllm-eval/scripts/build_verl_oven_parquet.py. wandb run: offline-run-20260704_175613-qwen3-vl-4b-oven-grpo-traversal-wikidata-shaped-clean-seed42.