Modelpublic

MSALab/PerceptionDLM

sourceHugging Faceapache-2.0updated 3mo agoView on Hugging Face

13likes69downloads

Model Card

PerceptionDLM

PerceptionDLM is a multimodal diffusion language model optimized for efficient parallel region perception. Built upon **PerceptionDLM-Base**, it fully leverages the parallel decoding nature of diffusion language models (DLMs): given an image and multiple region masks, it generates descriptions for all regions simultaneously within a single denoising process — avoiding the linear latency growth of autoregressive (AR) region captioners.

To the best of our knowledge, this is the first model to achieve parallel region captioning and perception by leveraging the advantages of diffusion language models.

<p align="center"> 📄 <a href="https://arxiv.org/abs/2606.19534">Paper</a>  |  💻 <a href="https://github.com/MSALab-PKU/PerceptionDLM">Code</a>  |  📊 <a href="https://huggingface.co/datasets/MSALab/ParaDLC-Bench">ParaDLC-Bench</a> </p>

Highlights

—🧩 Parallel region captioning. Region prompting + structured attention masking describe many masked regions in a single denoising pass.
—⚡ Up to 3.44× throughput speedup in dense multi-region scenarios, with stable per-image latency (~2.9s).
—🎯 Competitive quality with strong AR region captioners while being substantially faster.

Model Details


Base model	MSALab/PerceptionDLM-Base
Key modules	Region prompting, RoI-aligned feature replay, structured attention masking
Region prompts	up to 6 per image
Default inference	32 diffusion steps, generation length 32 per mask
Training	full ParaCaption corpus, ~2 days on 32× H100
Precision	bfloat16

Results (ParaDLC-Bench)

Method	Type	Avg (%)	TPF ↑	Time (s) ↓
GAR-8B	AR (sequential)	69.5	1.0	479
LLaDA-V-8B	Diffusion	35.2	1.0	3241
PerceptionDLM	Diffusion (parallel)	62.4	2.9	276

TPF = Tokens Per Forward (higher = more parallel). PerceptionDLM nearly doubles the accuracy of prior diffusion VLMs while drastically reducing inference time.

Usage

Full inference scripts are provided in the GitHub repository.

bash

python demo/infer_pdmllm.py \
  --model-path MSALab/PerceptionDLM \
  --image assets/demo.jpg \
  --masks assets/demo_mask_0.jpg \
          assets/demo_mask_1.jpg \
          assets/demo_mask_2.jpg \
  --gen-length 32 --steps 32 --temperature 0.0 --top-p 1.0

The model takes an RGB image plus one or more binary masks, and returns one caption per region — all generated in parallel.

Citation

bibtex

@article{sun2026perceptiondlm,
  title   = {PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models},
  author  = {Sun, Yueyi and Wang, Yuhao and Li, Jason and Tian, Ye and Zhang, Tao and Mai, Jacky and Wang, Yihan and Wang, Haochen and Bai, Jinbin and Yang, Ling and Tong, Yunhai},
  journal = {arXiv preprint arXiv:2606.19534},
  year    = {2026}
}

License

Released under the Apache License 2.0.