RLinf/RLinf-OpenVLA-GRPO-ManiSkill3-25ood

<div align="center"> <img src="logo.svg" alt="RLinf-logo" width="500"/> </div>

<div align="center"> <!-- <a href="TODO"><img src="https://img.shields.io/badge/arXiv-Paper-red?logo=arxiv"></a> --> <!-- <a href="TODO"><img src="https://img.shields.io/badge/HuggingFace-yellow?logo=huggingface&logoColor=white" alt="Hugging Face"></a> --> <a href="https://github.com/RLinf/RLinf"><img src="https://img.shields.io/badge/Github-blue"></a> <a href="https://rlinf.readthedocs.io/en/latest/"><img src="https://img.shields.io/badge/Documentation-Purple?color=8A2BE2&logo=readthedocs"></a> <!-- <a href="TODO"><img src="https://devin.ai/assets/deepwiki-badge.png" alt="Ask DeepWiki.com" style="height:20px;"></a> <a href="TODO"><img src="https://img.shields.io/badge/微信-green?logo=wechat&amp"></a> --> </div>

<h1 align="center">RLinf: Reinforcement Learning Infrastructure for Agentic AI</h1>

RLinf is a flexible and scalable open-source infrastructure designed for post-training foundation models (LLMs, VLMs, VLAs) via reinforcement learning. The 'inf' in RLinf stands for Infrastructure, highlighting its role as a robust backbone for next-generation training. It also stands for Infinite, symbolizing the system’s support for open-ended learning, continuous generalization, and limitless possibilities in intelligence development.

<div align="center"> <img src="overview.png" alt="RLinf-overview" width="600"/> </div>

Model Description

This model is trained on `gen-robot/openvla-7b-rlvla-warmup` by Group Relative Policy Optimization (GRPO) on the ManiSkill simulator.

Full OOD Evaluation and Results

Overall Eval Results

Note: rl4vla refers to the paper VLA-RL-Study: What Can RL Bring to VLA Generalization? An Empirical Study. | Description | rl4vla | GRPO-openvlaoft | _PPO-openvlaoft_ | PPO-openvla | GRPO-openvla | |---------------|-----------|-----------------|----------------|-------------|---------------| | Avg results | 0.7915 | 0.6064 | 0.7705 | 0.8193 | 0.7515 |

Training Setting Eval

OOD Eval on Vision

OOD Eval on Semantic

OOD Eval on Position

How to Use

Please integrate the provided model with the RLinf codebase. To do so, modify the following parameters in the configuration file `examples/embodiment/config/maniskill_grpo_openvla.yaml`:

• —Set `actor.checkpoint_load_path, actor.tokenizer.tokenizer_model, and rollout.model_dir` to the path of the model checkpoint.

Note: If you intend to evaluate the model directly, make sure to set `actor.model.is_lora to false`.

License

This code repository and the model weights are licensed under the MIT License.