Lordvarun23/qwen2.5-vl-3b-mllmu-manu-forget5

Qwen2.5-VL-3B MLLMU-Bench: forget_5 unlearned with MANU (Modality-Aware Neuron Unlearning)

`Lordvarun23/qwen2.5-vl-3b-mllmu-ft` (finetuned on all 500 fictitious profiles) after unlearning the 25 forget_5 profiles with MANU (Liu et al., ACL 2025). MANU is training-free: it zeroes the MLP neurons that are much more active on forget data than on retain data.

Algorithm

1. 1.Collect activations of every MLP intermediate neuron (the input of down_proj), streamed over tokens (pad tokens excluded):
2. 2.vision tower (32 blocks x 3,420 neurons) on multimodal inputs: image + question -> answer
3. 3.language model (36 layers x 11,008 neurons) on text-only inputs: "this person" replaced by the profile's name on the forget set and on an equally sized random sample of the retain set.
4. 4.Per-neuron importance statistics $I_k$ on each set: mean $|a|$ (abs), $P(|a|>0.1)$ (freq), $\mathrm{std}(a)$ (var), $\sqrt{\mathbb{E}[a^2]}$ (rms).
5. 5.Score (as in the official MANU code):

$$Sn = \sumk wk\left(\frac{Ik(\mathcal{D}f, n)}{Ik(\mathcal{D}_r, n) + \epsilon} - 1\right), \quad w = (\text{abs}{:}2,\ \text{freq}{:}0,\ \text{var}{:}2,\ \text{rms}{:}2)$$

1. 1.Prune the top α = 2% of neurons by $Sn$, globally within each group (vision / language model): zero the neuron's `gateproj and upproj` rows (and biases) and its `downproj` column.

Differences from the official code: every layer gets its own statistics (the official hooks pool all layers under the names of the last three layers and prune only those), pad tokens are excluded, and the person's name for the text-only inputs comes from the biography JSON instead of being parsed from QA answers.

Data

MLLMU-Bench contains 500 fictitious people. Each has a face image and 15-20 image-grounded QA pairs plus one biography answer (ft_Data). The benchmark splits the profiles into forget_5 (25 profiles) and retain_95 (475 profiles). Retain_Set holds 153 real celebrities and is used only to measure general knowledge (utility).

Unlearning setup

Evaluation

• —Task: MLLMU-Bench Generation_Task, 4 open-ended questions per profile (2 Image_Textual + 2 Pure_Text). Every question is asked together with the profile image. Split sizes: forget5 = 100 questions, retain95 = 1900, Retain_Set = 612.
• —Decoding: vLLM, greedy, max 128 new tokens, default Qwen system prompt, images resized to 128-256 x 28x28 pixels.
• —ROUGE: rouge_score with stemming, generation vs ground truth (ROUGE-L F1 reported).
• —LLM judge: Qwen/Qwen2.5-7B-Instruct, binary per answer: 1 = the answer contains the ground-truth fact or a semantically equivalent one, 0 = wrong, missing, partial or merely similar (e.g. a different city or salary). Judge accuracy is the mean over questions.

Goal of unlearning: forget_5 ↓ (toward the retain-only model), with retain_95 and Retain_Set close to the finetuned model.

The row in bold is this model. forget_5 has only 100 questions, so judge-accuracy differences below about 0.05 are within noise.

Summary: forget5 judge accuracy goes from 0.49 to **0.29** (retain-only reference: 0.13). retain95 goes from 0.63 to 0.47, and real-celebrity utility (RetainSet) goes from 0.20 to 0.15. The pruned neurons are partly shared general-knowledge features, so MANU costs more real-celebrity knowledge than the training-based methods. Larger ratios were also tried: α=5% gave forget/retain95/Retain_Set judge accuracy 0.23/0.34/0.14, and α=10% gave 0.08/0.23/0.08 (utility collapse).

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

repo = "Lordvarun23/qwen2.5-vl-3b-mllmu-manu-forget5"
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": "What is the name of this person?"}]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[Image.open("face.jpg")], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.batch_decode(out[:, inputs.input_ids.shape[1]:], skip_special_tokens=True)[0])

Limitations

• —Research artifact for studying machine unlearning. The people in MLLMU-Bench are fictitious, and this model will state made-up biographical facts about faces with confidence. Do not use it to identify real people or for factual answers about them.
• —Results come from a single seed and one checkpoint, without a hyperparameter search beyond the one described.
• —Released under the base model's Qwen Research License.