Lordvarun23/qwen2.5-vl-3b-mllmu-klmin-forget5

Qwen2.5-VL-3B MLLMU-Bench: forget5 unlearned with KL Minimization (KLMin)

`Lordvarun23/qwen2.5-vl-3b-mllmu-ft` (finetuned on all 500 fictitious profiles) after unlearning the 25 forget_5 profiles with KL Minimization (KLMin), regularized on the retain95 profiles.

Algorithm

KL Minimization: GradDiff plus a KL term on the retain batch that keeps the model's full next-token distribution close to the original finetuned model (MLLMU-Bench formulation, retain NLL + KL):

$$\mathcal{L} = -\,\mathrm{NLL}\theta(\text{forget}) + \lambda\Big[\mathrm{NLL}\theta(\text{retain}) + \mathrm{KL}\big(\pi{\text{ref}}\,\|\,\pi\theta\big)(\text{retain})\Big]$$

NLL and KL are both averaged over the answer tokens. $\pi_{\text{ref}}$ is a frozen copy of the starting finetuned model. The official MLLMU-Bench code scales the KL per sequence (batchmean over every token), which weights it more heavily.

Data

MLLMU-Bench contains 500 fictitious people. Each has a face image and 15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only to measure general knowledge (utility).

Unlearning setup

Evaluation

• —Task: MLLMU-Bench Generation_Task, 4 open-ended questions per profile (2 Image_Textual + 2 Pure_Text). Every question is asked together with the profile image. Split sizes: forget5 = 100 questions, retain95 = 1900, Retain_Set = 612.
• —Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
• —ROUGE: rouge_score with stemming, generation vs ground truth (ROUGE-L F1 reported).
• —LLM judge: Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.

Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.

The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.

Summary: forget5 judge accuracy goes from 0.49 to **0.32** (retain-only reference: 0.13). retain95 goes from 0.63 to 0.52, and real-celebrity utility (Retain_Set) goes from 0.20 to 0.17. Utility drops but does not collapse.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-klmin-forget5"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Limitations

• —Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
• —Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
• —Released under the base model's Qwen Research License.