MSALab/PerceptionDLM-Base

1---2license: apache-2.03language:4- en5library_name: transformers6pipeline_tag: image-text-to-text7base_model:8- MSALab/LLaDA-8B-Instruct-HF9tags:10- multimodal11- diffusion-language-model12- dllm13- vision-language-model14- perception15---16 17# PerceptionDLM-Base18 19**PerceptionDLM-Base** is a strong open **multimodal diffusion language model (DLM)** that extends a large language diffusion backbone (LLaDA-8B) to visual instruction tuning. It establishes a new state-of-the-art baseline among open discrete-diffusion VLMs, outperforming LLaDA-V on **15 / 16** standard multimodal benchmarks while remaining competitive with same-scale autoregressive (AR) VLMs.20 21It serves as the foundation model for [**PerceptionDLM**](https://huggingface.co/MSALab/PerceptionDLM), our parallel region-perception model.22 23<p align="center">24  📄 <a href="https://arxiv.org/abs/2606.19534">Paper</a> &nbsp;|&nbsp;25  💻 <a href="https://github.com/MSALab-PKU/PerceptionDLM">Code</a> &nbsp;|&nbsp;26  🤗 <a href="https://huggingface.co/collections/MSALab/perceptiondlm-model-zoo">Model Collection</a>27</p>28 29## Highlights30 31- 🧠 **Diffusion-based VLM.** Non-autoregressive masked-denoising generation with intrinsic token-level parallelism.32- 🏗️ **LLaVA-style architecture.** SigLIP-2 vision encoder + 2-layer MLP connector + LLaDA-8B diffusion decoder, with dynamic-resolution tiling for high-resolution inputs.33- 🏆 **Strong baseline.** Outperforms LLaDA-V on 15/16 benchmarks; especially strong on fine-grained perception and hallucination robustness.34 35## Model Details36 37| | |38| :--- | :--- |39| Vision encoder | `google/siglip2-so400m-patch16-512` (frozen) |40| Connector | 2-layer MLP with GELU |41| Language backbone | LLaDA-Instruct-8B (diffusion) |42| Parameters | ~8B |43| Training | 4-stage visual instruction tuning, 32× H100 (~3 weeks) |44| Precision | bfloat16 |45 46## Results47 48PerceptionDLM-Base vs. open diffusion / AR VLMs (selected benchmarks):49 50| Benchmark | PerceptionDLM-Base | LLaDA-V | Qwen2.5-VL-7B | InternVL3-8B |51| :--- | :---: | :---: | :---: | :---: |52| MMBench | **85.0** | 82.9 | 83.5 | 83.4 |53| SeedBench | **78.9** | 74.8 | 77.0 | 77.1 |54| ChartQA | **91.6** | 78.3 | 86.2 | 86.6 |55| MMVP | **82.0** | 76.7 | 73.3 | 80.0 |56| BLINK | **60.3** | 50.9 | 55.3 | 55.5 |57| RealWorldQA | **73.7** | 63.2 | 68.4 | 70.8 |58| HallusionBench | **58.4** | 50.9 | 51.9 | 49.9 |59 60See the [paper](https://arxiv.org/abs/2606.19534) for the full 16-benchmark comparison.61 62## Usage63 64Full inference scripts are provided in the [GitHub repository](https://github.com/MSALab-PKU/PerceptionDLM).65 66```bash67python demo/infer_dmllm.py \68  --model-path MSALab/PerceptionDLM-Base \69  --image assets/demo.jpg \70  --prompt "What color shirt is the man in the picture wearing?" \71  --gen-length 64 --block-length 64 --steps 6472```73 74```python75import torch76from transformers import AutoModel, AutoProcessor77 78model_path = "MSALab/PerceptionDLM-Base"79processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)80model = AutoModel.from_pretrained(81    model_path, torch_dtype=torch.bfloat16, trust_remote_code=True82).cuda().eval()83# See demo/infer_dmllm.py for the full preprocessing + generation pipeline.84```85 86## Citation87 88```bibtex89@article{sun2026perceptiondlm,90  title   = {PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models},91  author  = {Sun, Yueyi and Wang, Yuhao and Li, Jason and Tian, Ye and Zhang, Tao and Mai, Jacky and Wang, Yihan and Wang, Haochen and Bai, Jinbin and Yang, Ling and Tong, Yunhai},92  journal = {arXiv preprint arXiv:2606.19534},93  year    = {2026}94}95```96 97## License98 99Released under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).100