fudan-generative-ai/Bard-VL-B4-Mask-2B-Instruct

<h1 align="center">BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation</h1>

<p align="center"> <a href="https://github.com/cbyzju">Baoyou Chen</a>^1,3 · <a href="https://github.com/1ring2rta">Hanchen Xia</a>¹ · <a href="https://github.com/yhpengtu-rgb">Peng Tu</a>¹ · <a href="https://github.com/Theseus-427">Haojun Shi</a>¹ · <a href="https://github.com/AricGamma">Liwei Zhang</a>¹ · <a href="https://github.com/weihaosky">Weihao Yuan</a>⁴ · <a href="https://sites.google.com/site/zhusiyucs/home">Siyu Zhu</a>^1,2,3,† </p>

<p align="center"> ¹Shanghai Academy of AI for Science &nbsp;&nbsp;·&nbsp;&nbsp; ²Shanghai Innovation Institute &nbsp;&nbsp;·&nbsp;&nbsp; ³Fudan University &nbsp;&nbsp;·&nbsp;&nbsp; ⁴Nanjing University </p>

<p align="center"> 🤗 <a href="https://huggingface.co/fudan-generative-ai/Bard-VL-B4-Mask-2B-Instruct">Model</a> &nbsp;&nbsp;|&nbsp;&nbsp; 🏠 <a href="https://fudan-generative-vision.github.io/Bard-VL">Project Page</a> &nbsp;&nbsp;|&nbsp;&nbsp; 📑 <a href="https://huggingface.co/papers/2604.16514">Paper</a> &nbsp;&nbsp;|&nbsp;&nbsp; ✨ <a href="https://github.com/fudan-generative-vision/Bard-VL">Code</a> </p>

Bard-VL-B4-Mask-2B-Instruct

Bard-VL-B4-Mask-2B-Instruct is a 2B-class vision-language instruction model with masked discrete-diffusion decoding.

It is part of the Bard-VL family and is designed to bridge autoregressive and diffusion-style vision-language models through Progressive Block Merging (PBM) and Stage-Wise Distillation (SWD).

Compared with a standard autoregressive VLM release style, Bard-VL emphasizes:

• —parallel block-wise decoding instead of token-by-token generation
• —controllable response generation through blockwise denoising

✨ Highlights

• —Progressive Block Merging: Bard-VL increases the decoding block size progressively instead of jumping directly from autoregressive decoding to large-block diffusion.
• —Stage-Wise dVLM Distillation: Bard-VL distills from a small-block diffusion anchor in the same denoising regime, reducing autoregressive-to-diffusion transfer mismatch.
• —Packed Multimodal Attention Mask: the packed attention layout reuses shared multimodal context across clean and noisy branches to reduce redundant computation.
• —Mixed-Noise Training: Bard-VL combines masked-token and uniform token corruption to support both token completion and visible-token revision.

🧭 Method Structure

<p align="center"> <img src="./model.PNG" alt="Bard-VL method overview" width="100%"> </p>

<p align="center"> <em>Pipeline, block-wise attention mask, and mixed-noise scheduler used by Bard-VL.</em> </p>

📊 Evaluation Results

AutoRegressive Vision-Language Models

Diffusion Vision-Language Models

Bard-VL Converted from Qwen3-VL

🛠️ Environment

Make sure your environment is aligned with the repository requirements.txt:

python>=3.10
torch==2.8.0
torchvision==0.23.0
transformers==4.57.3
diffusers==0.36.0
accelerate==1.12.0
deepspeed==0.17.0

Recommended runtime settings in the local repository:

dtype = bfloat16
attn_implementation = sdpa
block_size = 4
denoising_steps = 4

🚀 Inference Example

The official repository inference flow is implemented in inference.py. A minimal image understanding example aligned with that script is shown below.

import torch
from transformers import AutoProcessor

from qwen_vl_utils import process_vision_info
from nemo_automodel.components.models.bard_vl import BardVLForConditionalGeneration

model_id = "fudan-generative-ai/Bard-VL-B4-Mask-2B-Instruct"
device = "cuda" if torch.cuda.is_available() else "cpu"

model = BardVLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    _attn_implementation="sdpa",
).to(device).eval()
processor = AutoProcessor.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": "You are a helpful assistant.",
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "assets/puzzle.jpg", "min_pixels": 256 * 256, "max_pixels": 2048 * 2048},
            {"type": "text", "text": "Please describe this image."},
        ],
    },
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

image_inputs, video_inputs, video_kwargs = process_vision_info(
    messages,
    return_video_kwargs=True,
    return_video_metadata=False,
    image_patch_size=processor.image_processor.patch_size,
)

batch = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=False,
    return_tensors="pt",
    **video_kwargs,
).to(device)

response_ids = model.generate(
    batch,
    max_new_tokens=1024,
    block_size=4,
    denoising_steps=4,
    temperature=0.0,
    top_k=0,
    top_p=1.0,
    remasking_strategy="low_confidence_dynamic",
    confidence_threshold=0.5,
    return_step_stats=False,
)

print(processor.tokenizer.batch_decode(response_ids, skip_special_tokens=True)[0].strip())

For video understanding, replace the image message with the video example in inference.py.

📚 Citation

@article{chen2026bard,
  title={BARD: Bridging AutoRegressive and Diffusion Vision-Language Models Via Highly Efficient Progressive Block Merging and Stage-Wise Distillation},
  author={Baoyou Chen and Hanchen Xia and Peng Tu and Haojun Shi and Liwei Zhang and Weihao Yuan and Siyu Zhu},
  journal={arXiv preprint arXiv:2604.16514},
  year={2026}
}