CoolFace
Modelpublic

chaitnya26/Lumina-DiMOO-fork

sourceHugging Faceapache-2.0updated 1y agoView on Hugging Face
0likes88downloads
Model Card

<p align="center"> <img src="./assets/Lumina-DiMOO.png" width="20%"/> </p>

<div align="center"> <h1> Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understanding </h1>

[[📑 Technical Report (Coming Soon)]()] &emsp; [💜 Project Page (Demo & Benchmark)] &emsp; [🤗 Model ]

<b>¹Shanghai AI Laboratory, ²Shanghai Innovation Institute, ³Shanghai Jiao Tong University</b>

<b>⁴Nanjing University, ⁵The University of Sydney</b>

<b>⁶The Chinese University of Hong Kong, ⁷Tsinghua University</b>

<img src="./assets/teaser.png" width="100%"/> </div>

📚 Introduction

We introduce Lumina-DiMOO, an omni foundational model for seamless multimodal generation and understanding. Lumina-DiMOO is distinguished by four key innovations:

  • —Unified Discrete Diffusion Architecture: Lumina-DiMOO sets itself apart from prior unified models by utilizing a fully discrete diffusion modeling to handle inputs and outputs across various modalities.
  • —Versatile Multimodal Capabilities: Lumina-DiMOO supports a broad spectrum of multimodal tasks, including text-to-image generation (allowing for arbitrary and high-resolution), image-to-image generation (e.g., image editing, subject-driven generation, and image inpainting, etc.), alongside advanced image understanding.
  • —Higher Sampling Efficiency: Compared to previous AR or hybrid AR-diffusion paradigms, Lumina-DiMOO demonstrates remarkable sampling efficiency. Additionally, we design a bespoke caching method to further speed up the sampling speed by 2x.
  • —Superior Performance: Lumina-DiMOO achieves state-of-the-art performance on multiple benchmarks, surpassing existing open-source unified multimodal models, setting a new standard in the field.

<img src="./assets/architecture.png" width="100%"/>

📽️ Qualitative Results

Here we present some comparative generation results with other models. For additional visualization results, please see our [Project Page](https://synbol.github.io/Lumina-DiMOO/). <details open> <summary>Text-to-Image Comparison</summary> <img src="./assets/demo_t2i.png" width="100%"/> </details>

<details close> <summary>Image Editing Comparison</summary> <img src="./assets/demo_editing.png" width="100%"/> </details>

<details close> <summary>Controllable & Subject-Driven Generation Comparison</summary> <img src="./assets/qualitativecontrolsubject.png" width="100%"/> </details>

<details close> <summary>Image Inpainting & Extrapolation</summary> <img src="./assets/demo_inpainting.jpg" width="100%"/> </details>

📊 Quantitative Performance

<details open> <summary>GenEval Benchmark</summary> <img src="./assets/GenEval_benchmark.png" width="100%"/> </details>

<details close> <summary>DPG Benchmark</summary> <img src="./assets/DPG_benchmark.png" width="100%"/> </details>

<details close> <summary>OneIG-EN Benchmark</summary> <img src="./assets/OneIG-EN_benchmark.png" width="100%"/> </details>

<details close> <summary>TIIF Benchmark</summary> <img src="./assets/TIIF_benchmark.png" width="100%"/> </details>

<details close> <summary>Image-to-Image Benchmark</summary> <img src="./assets/i2i_benchmark.png" width="100%"/> </details>

<details close> <summary>Image Understanding Benchmark</summary> <img src="./assets/understanding_benchmark.png" width="100%"/> </details>

🚀 Sampling Speed Analysis

  • —Since text generation is performed in a block-wise manner, unlike image generation which uses a single global decoding step, its speed is influenced by both the number of blocks and the number of steps. Therefore, the speed improvement of image understanding is not as significant as that of image generation.
  • —Lumina-DiMOO Settings: For image generation, we sample 64 steps. For image understanding, we set the block length to 256 and the number of sampling steps to 128.

<details open> <summary>Sampling Speed Comparison</summary> <img src="./assets/speed_comparison.png" width="100%"/> </details>

💬 Discussion

You can reach us with this WeChat QR code! <p align="left"> <img src="./assets/wechat.jpeg" width="50%"/> <br> </p>

📜 Acknowledgements

This work was also supported and implemented by MindSpeed MM, an open-source training framework for large-scale multimodal models designed for distributed training, developed and maintained by Huawei's Computing Product Line. Specifically Optimized for Huawei‘s Ascend AI chips, MindSpeed MM offers comprehensive support for distributed training and is tailored for a wide range of multimodal tasks.

📖 BibTeX

@misc{lumina-dimoo,
      title={Lumina-DiMOO: A Unified Masked Diffusion Model for Multi-Modal Generation and Understanding},
      author={Alpha VLLM Team},
      year={2025},
      url={https://github.com/Alpha-VLLM/Lumina-DiMOO},
}