HumorR1/policy-qwen3vl-2b-grpo-newyorker

humor-r1 SFT+GRPO policy — Qwen3-VL-2B-Thinking + LoRA

A captioning policy for New Yorker–style cartoons trained in two stages:

1. 1.SFT on 813 (cartoon, prompt, top-rated caption) triples augmented with synthetic chain-of-thought traces (the base model's "explain why this caption is funny" given the answer).
2. 2.GRPO on the SFT'd policy against a Bradley-Terry reward model (`HumorR1/rm-qwen25vl-3b-20k`). Reward = sigmoid(RM_score) for completions matching <caption>...</caption>, else 0.

The output format is <think>...brief reasoning...</think>\n\n<caption>X</caption>.

Headline result

50 cartoons × 5 samples = 235 generations per policy. Eval uses max_new_tokens=8192 — the chain-of-thought is verbose (median ~2500 tokens), so smaller budgets clip most completions and make the metric meaningless. Use this same budget at inference.

The SFT-only intermediate (no thinking) scored emitted RM -1.74 (stronger raw quality, but no chain-of-thought). This policy keeps the thinking format the project proposal calls for and beats base by +1.69σ on emitted RM while completing captions ~7 pp more reliably (96% vs 89%).

Training pipeline (in order)

1. 1.scripts/synthesize_thinking.py — base Qwen3-VL-2B-Thinking generates a brief explanation for each (cartoon, top-rated caption) pair. Saved as data/caption_sft_train_with_thinking.
2. 2.scripts/train_sft.py — SFT for 2 epochs on the augmented dataset. LoRA r=α=32, lr=2e-4 constant, no warmup, eff_bs=32, bf16, FA2. Target = {thinking}</think>\n\n<caption>{caption}</caption>.
3. 3.Merge SFT LoRA into base, save as qwen3vl-2b-sft-think-merged.
4. 4.scripts/train_grpo_qwen3vl.py with POLICY_MODEL_NAME=...sft-think-merged, FORMAT_BONUS=0.0, MAX_STEPS=200, NUM_GENERATIONS=8, MAX_COMPLETION_LENGTH=1536. Single A100, vLLM colocate.

Sample outputs

Held-out cartoons (RM scores in parentheses, higher is better):

Contest 543 (cartoon: elephant in a living room; people ignore it) - base (-4.66): "The two people in the living room try to read while the elephant stands there, like it's the secret probably lost." - policy (-0.89): "Elephant's in the living room, but the humans are still reading."

Contest 568 (cartoon: woman in glass talks to two people stuck in buckets) - base (-3.27): "At the party, you're supposed to sip, but you're bound in buckets—no toast, just a bucket-and-meeting!" - policy (-0.90): "At the party where the only social event that works is standing in buckets."

Limitations

• —Trained on the New Yorker humor distribution; OOD on other cartoons unverified.
• —The thinking traces are synthetic (base model's post-hoc explanations of good captions). They give the policy a thinking habit but they don't embody true generative reasoning — they're closer to "warmups" than reasoning steps that find the punchline. A 7B+ model with longer training would likely benefit more from real CoT.
• —Format rate of 0.33 means most generations don't emit a caption inside the 2048-token max. Use vLLM with stop=["</caption>"] and rely on best-of-N sampling at inference — best-of-3 should hit a valid caption in >70% of cartoons.
• —Reward is calibrated against an RM with 0.682 pairwise accuracy on held-out preference pairs.

Usage

For best throughput, vLLM with enable_lora=True and a LoRARequest pointing at this adapter (see scripts/eval_policy.py in the source repo).