Modelpublic

wudq/EmObserver

sourceHugging Faceapache-2.0updated 2mo agoView on Hugging Face

0likes12downloads

Model Card

EmObserver

Official model release for “MVEI & EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement.”

EmObserver is an emotion-oriented multimodal large language model built from Qwen3-VL-8B-Thinking. It is optimized through a four-stage training recipe for visual emotion understanding, Emotion Statement Judgement (ESJ), emotion classification, and single-/multi-image affective reasoning.

The model accepts one or more images with a natural-language question and produces a reasoning-style response followed by a concise answer:

text

<think>...</think><answer>...</answer>

🔗 Project map

Resource	Description	Link
Paper	MVEI & EmObserver	arXiv:2607.21061
Original paper (ICLR 2026)	Customizing Visual Emotion Evaluation for MLLMs	OpenReview · arXiv:2509.21950
Official code	Inference, evaluation, and four-stage training	wdqqdw/EmObserver
Expanded release	Benchmarks, images, predictions, and training data	wudq/MVEI_PLUS
EmObserver results	Released predictions and metric summaries	MVEI_PLUS/baselines/Qwen3-VL-EMOBSERVER
Evaluation data	Standardized MVEI, EEmo-Bench, and VECBench metadata	MVEI_PLUS/benchmarks
Training data	Four-stage data and GPT-5.5-filtered INSETS-462k	MVEI_PLUS/training_data
Original MVEI dataset	Original benchmark release	wudq/MVEI
Original INSETS-462k	Original training-data release	wudq/INSETS-462k
Original conference code	Earlier MVEI codebase and project release	wdqqdw/MVEI
Base model	Qwen3-VL-8B-Thinking	Qwen/Qwen3-VL-8B-Thinking

🚀 Quick start with Transformers

Install a compatible PyTorch build first, then install the model dependencies:

bash

python3 -m pip install -U "transformers>=5.9.0" accelerate pillow

The following example uses the same MVEI question form as the released test metadata:

python

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "wudq/EmObserver"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

image_url = (
    "https://huggingface.co/datasets/wudq/MVEI_PLUS/resolve/main/"
    "benchmarks/MVEI/images/contentment/contentment_14236.jpg"
)
prompt = (
    "Is the following statement correct about the image? Upon viewing this image, "
    "observers, despite various individual or contextual factors, are most likely "
    "to experience negative emotions. Choose the answer from "
    "{'A': 'Correct.', 'B': 'Incorrect.'}. Answer in the format of "
    "<think>...</think><answer>...</answer>."
)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": image_url},
            {"type": "text", "text": prompt},
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated = model.generate(**inputs, max_new_tokens=1024)
trimmed = [output[len(input_ids):] for input_ids, output in zip(inputs.input_ids, generated)]
response = processor.batch_decode(
    trimmed,
    skip_special_tokens=True,
    clean_up_tokenization_spaces=False,
)[0]
print(response)

For multi-image questions, add each image to content before the text item:

python

"content": [
    {"type": "image", "url": first_image_url},
    {"type": "image", "url": second_image_url},
    {"type": "text", "text": pair_prompt},
]

FlashAttention 2 can reduce memory use on supported hardware. Install a build compatible with your CUDA/PyTorch stack, then pass attn_implementation="flash_attention_2" to from_pretrained.

🧪 Full evaluation with the official code

The official repository provides the vLLM inference wrapper, benchmark loaders, output parsing, and metric calculators:

bash

git clone https://github.com/wdqqdw/EmObserver.git
cd EmObserver

python3 -m pip install -r requirements.txt
hf download wudq/MVEI_PLUS --repo-type dataset --local-dir public_data
hf download wudq/EmObserver --local-dir models/EmObserver

python3 -m evaluate.infer_and_eval \
  --engine qwen3_vl_vllm \
  --size emobserver \
  --gpu_n 8

Adjust --gpu_n to the visible GPUs that can hold the checkpoint. Use --bench to select a subset from MVEI, VECBench, EEmo-Bench-Single-Perception, and EEmo-Bench-Pair-Perception.

💬 Benchmark-style examples

The examples below reproduce question forms from the standardized test metadata. The listed answer is the reference final answer; reasoning text may vary.

1. Emotion Statement Judgement — MVEI

Image: contentment_14236.jpg

text

Is the following statement correct about the image? Upon viewing this image,
observers, despite various individual or contextual factors, are most likely
to experience negative emotions. Choose the answer from
{'A': 'Correct.', 'B': 'Incorrect.'}. Answer in the format of
<think>...</think><answer>...</answer>.

Reference final answer:

text

<answer>Incorrect.</answer>

2. Single-image emotion perception — EEmo-Bench

Image: EEmo-Bench single image 1

text

What do you think of surprise as one of the three main emotions you felt from
this image? Choose the answer from {'A': 'No', 'B': 'Yes'}. Answer in the
format of <think>...</think><answer>...</answer>.

Reference final answer:

text

<answer>Yes.</answer>

3. Image-pair emotion comparison — EEmo-Bench

Images: image 579 and image 100

text

Which emotion is evoked in both images and is common to them? Choose the
answer from {'A': 'Neutral', 'B': 'Fear', 'C': 'Sadness', 'D': 'Joy'}.
Answer in the format of <think>...</think><answer>...</answer>.

Reference final answer:

text

<answer>Joy.</answer>

4. Visual emotion classification — VECBench

Image: abstract_0001.jpg

text

Which emotion might this image evoke? Choose the most likely one from
['Amusement', 'Anger', 'Awe', 'Content', 'Disgust', 'Excitement', 'Fear',
'Sad']. Think step by step. Respond in the format:
<think>{your reasoning}</think><answer>{your final answer}</answer>.

Reference final answer:

text

<answer>Content.</answer>

📊 Released evaluation results

These values are taken from the released EmObserver inference outputs in MVEI_PLUS. They are intended to make the checkpoint release traceable; environment or decoding changes may produce different results.

Benchmark	Metric	Result
MVEI	Overall accuracy	86.23%
MVEI	Sentiment polarity	86.47%
MVEI	Emotion interpretation	83.55%
MVEI	Scene context	90.30%
MVEI	Perception subjectivity	86.06%
VECBench	Overall accuracy	63.42%
EEmo-Bench Single	Overall accuracy	71.94%
EEmo-Bench Pair	Overall accuracy	71.47%

🏋️ Training overview

EmObserver is produced by the four-stage public training pipeline:

1.supervised fine-tuning;
2.GRPO-based reinforcement learning;
3.on-policy self-distillation (OPSD);
4.GRPO with an LLM-consistency reward.

The runnable scripts are in the official code repository, and their released inputs are under MVEI_PLUS/training_data. External judge credentials are not included; users must provide their own endpoint when reproducing the final stage.

⚠️ Intended use and limitations

EmObserver is intended for research on visual emotion understanding, affective image analysis, multimodal reasoning, and benchmark development.

—Visual emotion is inherently subjective and culturally/contextually dependent. A model prediction should not be treated as a universal description of how every person will feel.
—The model may inherit social, cultural, demographic, and content biases from its base model and training data.
—The checkpoint is not designed for mental-health diagnosis, psychological profiling, surveillance, hiring, education assessment, or other high-stakes decisions about individuals.
—Generated reasoning can be plausible but factually or emotionally incorrect. Verify outputs when reliability matters.
—Users are responsible for reviewing the licenses, attribution requirements, privacy constraints, and redistribution terms of input images and downstream datasets.

📄 License

The checkpoint is released under the Apache 2.0 license, following its Qwen3-VL-8B-Thinking base model. Third-party datasets and images retain their respective licenses and terms.

📝 Citation

If you use EmObserver or the expanded training and evaluation release, please cite the EmObserver paper below. When using the MVEI benchmark, dataset, or Emotion Statement Judgement formulation, please also cite the original ICLR 2026 paper.

bibtex

@article{wu2026mvei_emobserver,
  title   = {MVEI \& EmObserver: Empowering MLLM-Oriented Visual Emotional Intelligence via Emotion Statement Judgement},
  author  = {Wu, Daiqing and Yang, Dongbao and Yao, Jiashu and Zhang, Hongrui and Ma, Can and Zhou, Yu and Zhao, Sicheng},
  journal = {arXiv preprint arXiv:2607.21061},
  year    = {2026}
}

@inproceedings{wu2026customizing,
  title     = {Customizing Visual Emotion Evaluation for MLLMs: An Open-vocabulary, Multifaceted, and Scalable Approach},
  author    = {Wu, Daiqing and Yang, Dongbao and Zhao, Sicheng and Ma, Can and Zhou, Yu},
  booktitle = {International Conference on Learning Representations},
  year      = {2026},
  url       = {https://openreview.net/forum?id=dQTSXWqZws}
}