IffYuan/Embodied-R1.5

1---2base_model:3- Qwen/Qwen3-VL-8B-Instruct4language:5- en6- zh7library_name: transformers8license: apache-2.09pipeline_tag: robotics10tags:11- embodied-ai12- vision-language-model13- embodied-reasoning14- spatial-reasoning15- pointing16- vla17- qwen3-vl18---19 20# Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models21 22<p align="center">23  🌐 <a href="https://embodied-r.github.io/">Project Page</a> &nbsp;|&nbsp;24  💻 <a href="https://github.com/pickxiguapi/Embodied-R1.5">Code</a> &nbsp;|&nbsp;25  🧰 <a href="https://github.com/pickxiguapi/EmbodiedEvalKit">EmbodiedEvalKit</a> &nbsp;|&nbsp;26  🤗 <a href="https://huggingface.co/collections/IffYuan/embodied-r15">Models & Datasets</a> &nbsp;|&nbsp;27  📄 <a href="https://huggingface.co/papers/2606.11324">Paper</a>28</p>29 30> *"Reasoning initiates the action; Action fulfills the reasoning."* — Wang Yangming (1509)31 32## Overview33 34**Embodied-R1.5** is a unified **Embodied Foundation Model (EFM)**, built on **Qwen3-VL-8B-Instruct**, that integrates comprehensive embodied reasoning within a single architecture. Building on [Embodied-R1](https://github.com/pickxiguapi/Embodied-R1), it leaps from a pointing specialist to a comprehensive EFM unifying **three core capabilities**:35 36- **Spatial cognition & reasoning** — comprehend the semantic and spatial structure of the physical world, including static geometric relations and dynamic interaction possibilities.37- **Task planning & correction** — cover the full task life cycle: long-horizon decomposition, next-step planning, process detection, error localization, and correction.38- **Embodied pointing & location** — ground high-level reasoning in coordinates and trajectories, covering referring expression grounding, region-level localization, functional (affordance) grounding, and visual trace generation.39 40Trained on a 15B-token corpus with a multi-task balanced RL recipe, it further drives a **Planner-Grounder-Corrector (PGC)** closed-loop framework where one model acts as planner, grounder, and corrector to autonomously complete long-horizon real-world tasks.41 42## Output Conventions43 44Embodied-R1.5 follows the Qwen3-VL chat format and outputs structured answers inside `<answer>...</answer>` tags. The supported task types and their answer formats are:45 46| Task Type | Answer Format (inside `<answer>`) |47|-----------|-----------------------------------|48| `multiple choice` | `A` |49| `numerical` | `3.14` |50| `open-ended` | free text |51| `math` | `$$-\dfrac{3}{2}$$` |52| `spatial grounding` | `{"boxes": [35, 227, 437, 932]}` |53| `point` | ` ```json54[{"point_2d": [230, 138]}]55``` ` |56| `trace` | ` ```json57[{"point_2d": [624, 469]}, ...]58``` ` |59| `trace_3d` | ` ```json60[{"point_2d": [463, 599], "depth": 1.08}, ...]61``` ` |62 63> **Coordinate & unit conventions.** All points (`point_2d`) and boxes are normalized to the `[0, 1000]` range, regardless of the original image resolution. For `trace_3d`, the `depth` value is in meters.64 65## Quick Start66 67```python68from transformers import AutoModelForImageTextToText, AutoProcessor69from PIL import Image70 71model_id = "IffYuan/Embodied-R1.5"72model = AutoModelForImageTextToText.from_pretrained(73    model_id, torch_dtype="auto", device_map="auto"74)75processor = AutoProcessor.from_pretrained(model_id)76 77image = Image.open("scene.jpg")78messages = [79    {80        "role": "user",81        "content": [82            {"type": "image"},83            {"type": "text", "text": "You are a robot performing manipulation tasks. "84                                     "The task instruction is: move the blue cube on top of the yellow cube. "85                                     "Use 2D points to mark the target location."},86        ],87    }88]89 90text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)91inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)92out = model.generate(**inputs, max_new_tokens=512)93print(processor.batch_decode(out, skip_special_tokens=True)[0])94```95 96The model reasons over the visual observation and emits its final decision within an `<answer>` tag, e.g. `<answer>[{"point_2d": [750, 748]}]</answer>`.97 98## Inference with vLLM (Recommended)99 100For higher throughput, serve the model with vLLM:101 102```bash103vllm serve IffYuan/Embodied-R1.5 \104  --served-model-name "Embodied-R1.5" \105  --tensor-parallel-size 1 \106  --mm-encoder-tp-mode data \107  --gpu-memory-utilization 0.7 \108  --async-scheduling \109  --media-io-kwargs '{"video": {"num_frames": 32}, "image": {"max_num": 32}}' \110  --max_model_len 20000 \111  --limit-mm-per-prompt '{"image": 8, "video": 1}' \112  --host 0.0.0.0 --port 22002113```114 115## Evaluation116 117For benchmark evaluation, see [EmbodiedEvalKit](https://github.com/pickxiguapi/EmbodiedEvalKit), an evaluation framework covering 25+ embodied benchmarks.118 119## Training & Data120 121Embodied-R1.5 is trained in two stages: SFT (LLaMA-Factory) followed by RFT (EasyR1). Full training scripts are available in the [GitHub repository](https://github.com/pickxiguapi/Embodied-R1.5). Datasets are released in the [Embodied-R1.5 HuggingFace collection](https://huggingface.co/collections/IffYuan/embodied-r15).122 123## Citation124 125If you find Embodied-R1.5 useful in your research, please cite:126 127```bibtex128@article{yuan2026embodiedr15,129  title={Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models},130  author={Yuan, Yifu and Huang, Yaoting and Yao, Xianze and Li, Yutong and Zhang, Shuoheng and Han, Linqi and Li, Pengyi and Sun, Jiangeng and Jia, Wenting and Zhao Zhang and Liu, Yuhao and Liao, Ruihao and Hu, Yucheng and Wu, Qiyu and Li, Yuxiao and Dong, Zibin and Ni, Fei and Zheng, Yan and Gu, Shuyang and Ma, Yi and Tang, Hongyao and Hu, Han and Hao, Jianye},131  journal={arXiv preprint},132  year={2026}133}134 135@article{yuan2025embodied,136  title={Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation},137  author={Yuan, Yifu and Cui, Haiqin and Huang, Yaoting and Chen, Yibin and Ni, Fei and Dong, Zibin and Li, Pengyi and Zheng, Yan and Hao, Jianye},138  journal={ICLR 2026},139  year={2025}140}141```142 143## License144 145Released under the Apache 2.0 license.