JreamH/CGMLLM

CG-MLLM (v0.1)

CG-MLLM: Captioning and Generating 3D Content via Multi-modal Large Language Models (ICML 2026)

CG-MLLM is a 3D multimodal large language model (3D MLLM) built upon Qwen3-VL and Hunyuan3D-2.1 VAE for unified 3D understanding, 3D captioning, and high-resolution 3D content generation.

It uses a Mixture-of-Transformer design: a TokenAR Transformer for token-level content and a BlockAR Transformer for block-level 3D latents, enabling long-context interaction between standard tokens and spatial blocks in one architecture.

Links

• —Paper (arXiv): https://arxiv.org/abs/2601.21798
• —Hugging Face Papers: https://huggingface.co/papers/2601.21798
• —Project page: https://cv.jream.top/CG-MLLM-page/
• —Code: https://github.com/dreaming-huang/CG-MLLM
• —ICML poster: https://icml.cc/virtual/2026/poster/63909

Model Details

Repository Contents

How to Use

Install and run inference from the official code repository:

git clone https://github.com/dreaming-huang/CG-MLLM.git
cd CG-MLLM
# follow Installation in the GitHub README, then:
hf download JreamH/CGMLLM ema.safetensors --local-dir models/CGMLLM

Example (image-to-3D):

python inference.py \
  --llm_base_path Qwen/Qwen3-VL-2B-Instruct \
  --checkpoint models/CGMLLM \
  --obj_vae_path tencent/Hunyuan3D-2.1 \
  --obj_vae_len 4096 \
  --use_qwen_vit --use_qwen_vl --qk_norm \
  --mode i2obj --image examples/chairo.png

See the GitHub README for text-to-3D, image understanding, and 3D understanding examples.

Citation

If you find this work useful, please cite:

@misc{huang2026cgmllmcaptioninggenerating3d,
      title={CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models},
      author={Junming Huang and Chi Wang and Letian Li and Guangkai Xu and Donglin Huang and Hao Chen and Qiang Dai and Weiwei Xu},
      year={2026},
      eprint={2601.21798},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2601.21798},
}

Acknowledgments

Inference code builds on BAGEL and the Hunyuan3D-2.1 shape VAE. Please follow their licenses when using those components.