IffYuan/Embodied-R1.5-8B-SFT

1---2license: apache-2.03language:4- en5- zh6pipeline_tag: image-text-to-text7library_name: transformers8tags:9- embodied-ai10- robotics11- vision-language-model12- embodied-reasoning13- spatial-reasoning14- pointing15- vla16- qwen3-vl17base_model:18- Qwen/Qwen3-VL-8B-Instruct19---20 21# Embodied-R1.5-8B-SFT: Evolving Physical Intelligence via Embodied Foundation Models22 23<p align="center">24  🌐 <a href="https://embodied-r.github.io/">Project Page</a> &nbsp;|&nbsp;25  💻 <a href="https://github.com/pickxiguapi/Embodied-R1.5">Code</a> &nbsp;|&nbsp;26  🧰 <a href="https://github.com/pickxiguapi/EmbodiedEvalKit">EmbodiedEvalKit</a> &nbsp;|&nbsp;27  🤗 <a href="https://huggingface.co/collections/IffYuan/embodied-r15">Models & Datasets</a>28</p>29 30> *"Reasoning initiates the action; Action fulfills the reasoning."* — Wang Yangming (1509)31 32> **This is the SFT checkpoint.** `Embodied-R1.5-8B-SFT` is the **Stage 1 supervised fine-tuning (SFT)** model of Embodied-R1.5, trained on the full embodied data corpus before reinforcement fine-tuning. It serves as the starting point for Stage 2 RFT. For the final RFT model, see [**IffYuan/Embodied-R1.5**](https://huggingface.co/IffYuan/Embodied-R1.5).33 34## Overview35 36**Embodied-R1.5** is a unified **Embodied Foundation Model (EFM)**, built on **Qwen3-VL-8B-Instruct**, that integrates comprehensive embodied reasoning within a single architecture. Building on our prior work [Embodied-R1](https://github.com/pickxiguapi/Embodied-R1), it leaps from a pointing specialist to a comprehensive EFM unifying **three core capabilities**:37 38- **Spatial cognition & reasoning** — comprehend the semantic and spatial structure of the physical world, including static geometric relations and dynamic interaction possibilities.39- **Task planning & correction** — cover the full task life cycle: long-horizon decomposition, next-step planning, process detection, error localization, and correction.40- **Embodied pointing & location** — ground high-level reasoning in coordinates and trajectories, covering referring expression grounding, region-level localization, functional (affordance) grounding, and visual trace generation.41 42Trained on a 15B-token corpus with a multi-task balanced RL recipe, it further drives a **Planner-Grounder-Corrector (PGC)** closed-loop framework where one model acts as planner, grounder, and corrector to autonomously complete long-horizon real-world tasks. With only 8B parameters, Embodied-R1.5 is best on **16 of 24** embodied VLM benchmarks (avg. **70.4%**), surpassing Gemini-Robotics-ER-1.5 and GPT-5.4; with light action-data fine-tuning it adapts into **Embodied-R1.5-VLA**, outperforming strong baselines like $\pi_{0.5}$ across 4 manipulation benchmark suites; and it generalizes zero-shot to real robots on instruction following, affordance grounding, articulated manipulation, and long-horizon tasks.43 44## Output Conventions45 46Embodied-R1.5 follows the Qwen3-VL chat format and outputs structured answers inside `<answer>...</answer>` tags. The supported task types and their answer formats are:47 48| Task Type | Answer Format (inside `<answer>`) |49|-----------|-----------------------------------|50| `multiple choice` | `A` |51| `numerical` | `3.14` |52| `open-ended` | free text |53| `math` | `$$-\dfrac{3}{2}$$` |54| `spatial grounding` | `{"boxes": [35, 227, 437, 932]}` |55| `point` | ` ```json\n[{"point_2d": [230, 138]}]\n``` ` |56| `trace` | ` ```json\n[{"point_2d": [624, 469]}, ...]\n``` ` |57| `trace_3d` | ` ```json\n[{"point_2d": [463, 599], "depth": 1.08}, ...]\n``` ` |58 59> **Coordinate & unit conventions.** All points (`point_2d`) and boxes are normalized to the `[0, 1000]` range, regardless of the original image resolution. For `trace_3d`, the `depth` value is in meters.60 61## Quick Start62 63```python64from transformers import AutoModelForImageTextToText, AutoProcessor65from PIL import Image66 67model_id = "IffYuan/Embodied-R1.5-8B-SFT"68model = AutoModelForImageTextToText.from_pretrained(69    model_id, torch_dtype="auto", device_map="auto"70)71processor = AutoProcessor.from_pretrained(model_id)72 73image = Image.open("scene.jpg")74messages = [75    {76        "role": "user",77        "content": [78            {"type": "image"},79            {"type": "text", "text": "You are a robot performing manipulation tasks. "80                                     "The task instruction is: move the blue cube on top of the yellow cube. "81                                     "Use 2D points to mark the target location."},82        ],83    }84]85 86text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)87inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)88out = model.generate(**inputs, max_new_tokens=512)89print(processor.batch_decode(out, skip_special_tokens=True)[0])90```91 92The model reasons over the visual observation and emits its final decision within an `<answer>` tag, e.g. `<answer>[{"point_2d": [750, 748]}]</answer>`.93 94## Inference with vLLM (Recommended)95 96For higher throughput, serve the model with vLLM:97 98```bash99vllm serve IffYuan/Embodied-R1.5-8B-SFT \100  --served-model-name "Embodied-R1.5-8B-SFT" \101  --tensor-parallel-size 1 \102  --mm-encoder-tp-mode data \103  --gpu-memory-utilization 0.7 \104  --async-scheduling \105  --media-io-kwargs '{"video": {"num_frames": 32}, "image": {"max_num": 32}}' \106  --max_model_len 20000 \107  --limit-mm-per-prompt '{"image": 8, "video": 1}' \108  --host 0.0.0.0 --port 22002109```110 111More runnable examples (vLLM online / offline, HuggingFace, point decoding & visualization) are provided in the [GitHub repository](https://github.com/pickxiguapi/Embodied-R1.5) under `inference/`.112 113## Evaluation114 115For benchmark evaluation, see [EmbodiedEvalKit](https://github.com/pickxiguapi/EmbodiedEvalKit), our evaluation framework covering 25+ embodied benchmarks.116 117## Training & Data118 119Embodied-R1.5 is trained in two stages: SFT (LLaMA-Factory) followed by RFT (EasyR1). This checkpoint is the **Stage 1 SFT** model. Full training scripts are available in the [GitHub repository](https://github.com/pickxiguapi/Embodied-R1.5). Datasets are released in the [Embodied-R1.5 HuggingFace collection](https://huggingface.co/collections/IffYuan/embodied-r15).120 121## Citation122 123If you find Embodied-R1.5 useful in your research, please cite our work:124 125```bibtex126@article{yuan2026embodiedr15,127  title={Embodied-R1.5: Evolving Physical Intelligence via Embodied Foundation Models},128  author={Yuan, Yifu and Huang, Yaoting and Yao, Xianze and Zhang, Shuoheng and Han, Linqi and Li, Yutong and Li, Pengyi and Sun, Jiangeng and Jia, Wenting and Hu, Yucheng and Liu, Yuhao and Liao, Ruihao and Wu, Qiyu and Li, Yuxiao and Zhang, Zhao and Dong, Zibin and Ni, Fei and Zheng, Yan and Gu, Shuyang and Ma, Yi and Tang, Hongyao and Hu, Han and Hao, Jianye},129  journal={arXiv preprint},130  year={2026}131}132 133@article{yuan2025embodied,134  title={Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation},135  author={Yuan, Yifu and Cui, Haiqin and Huang, Yaoting and Chen, Yibin and Ni, Fei and Dong, Zibin and Li, Pengyi and Zheng, Yan and Hao, Jianye},136  journal={ICLR 2026},137  year={2025}138}139 140@article{yuan2025seeing,141  title={From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation},142  author={Yuan, Yifu and Cui, Haiqin and Chen, Yibin and Dong, Zibin and Ni, Fei and Kou, Longxin and Liu, Jinyi and Li, Pengyi and Zheng, Yan and Hao, Jianye},143  journal={ICLR 2026},144  year={2025}145}146```147 148## License149 150Released under the Apache 2.0 license.151