Modelpublic

waybarrios/MoDA-LLaVA-MoRE-8B-SigLIP-S2

sourceHugging Facellama3.1updated 3mo agoView on Hugging Face

0likes9downloads

Model Card

MoDA — LLaVA-MoRE 8B (SigLIP-SO400M + S2)

Official checkpoint for "MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs" (ICML 2026).

📄 Paper: hf.co/papers/2506.01850 · arXiv:2506.01850 💻 Code: github.com/waybarrios/MoDA 🌐 Project page: waybarrios.com/MoDA

MoDA (Modulation Adapter) is a lightweight module that improves fine-grained visual grounding in Multimodal LLMs through instruction-guided, channel-wise modulation of pre-aligned visual features. A stack of cross-attention layers conditions on the language instruction and produces a soft channel-wise mask (via sigmoid) that is applied multiplicatively (Hadamard product) to the aligned visual features before they reach the LLM:

Ṽ_aligned = V_aligned ⊙ σ(W · F(T, V_aligned))

MoDA adds only <1% FLOPs and 3.7% parameters, requires no extra data or supervision, and plugs into the standard two-stage LLaVA instruction-tuning pipeline.

Model details


Architecture	LLaVA-MoRE (`LlavaLlamaForCausalLM`) + MoDA adapter (2 cross-attention decoder layers, 16 heads — paper config)
LLM backbone	meta-llama/Llama-3.1-8B-Instruct
Vision encoder	google/siglip-so400m-patch14-384 with S2 multi-scale features (384 / 768 / 1152)
Projector	2-layer MLP (GELU)
Training data	LLaVA v1.5 instruction-tuning mix (665K)
Training recipe	Two-stage LLaVA protocol: (1) projector pretraining; (2) MoDA + LLM joint fine-tuning
Precision	bfloat16

Results

Selected results for this checkpoint (LLaVA-MoRE SigLIP-S2 backbone; see the paper for the full 12-benchmark evaluation):

Benchmark	Baseline	+ MoDA
ScienceQA	77.1	81.9 (+4.8)
POPE	86.0	87.7

Gains are strongest on fine-grained, vision-centric, and hallucination tasks.

Usage

This checkpoint uses the LLaVA-MoRE + MoDA codebase (not vanilla transformers):

bash

git clone https://github.com/waybarrios/MoDA.git
cd MoDA/llava-more
pip install -r requirements.txt

python

from llava.model.builder import load_pretrained_model
from llava.mm_utils import get_model_name_from_path

model_path = "waybarrios/MoDA-LLaVA-MoRE-8B-SigLIP-S2"
tokenizer, model, image_processor, context_len = load_pretrained_model(
    model_path=model_path,
    model_base=None,
    model_name=get_model_name_from_path(model_path),
)

See the repository README for training instructions.

Evaluation

We evaluate with lmms-eval, which provides standardized implementations of all benchmarks reported in the paper (GQA, ScienceQA, POPE, MMBench, RealWorldQA, and more).

Citation

bibtex

@inproceedings{barrios2026moda,
  title     = {MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs},
  author    = {Barrios, Wayner and Villa, Andr\'es and Leon Alcazar, Juan C. and Jin, SouYoung and Ghanem, Bernard},
  booktitle = {Proceedings of the International Conference on Machine Learning (ICML)},
  year      = {2026}
}

License

The MoDA code is released under the MIT License. This checkpoint is a derivative of Llama 3.1 and is distributed under the Llama 3.1 Community License.

Acknowledgments

Built on LLaVA and aimagelab/LLaVA-MORE. Supported by startup funds from Dartmouth College and by KAUST — Center of Excellence for Generative AI (award 5940).