yqi19/finetune_xvla

🤖 X-VLA: Soft-Prompted Transformer as a Scalable Cross-Embodiment Vision-Language-Action Model

🏆 Highlights & News

🎉 Exciting News: X-VLA Accepted to ICLR 2026

We are thrilled to announce that X-VLA has been accepted to ICLR 2026.

🚀 Now Supported in LeRobot

X-VLA is now natively integrated into the LeRobot platform. Give it a try! We sincerely appreciate the support and collaboration from the Hugging Face team.

🥇 Champion Winner at IROS 2025

X-VLA won 1st Place (Champion) at the AgiBot World Challenge, held at IROS 2025.

🧩 Overview

Successful generalist Vision–Language–Action (VLA) models depend on scalable, cross-platform training across diverse robotic embodiments. To leverage the heterogeneity of large-scale robot datasets, X-VLA introduces a soft prompt mechanism — embodiment-specific learnable embeddings that guide a unified Transformer backbone toward effective multi-domain policy learning.

The resulting architecture — X-VLA-0.9B — achieves state-of-the-art generalization across six simulation platforms and three real-world robots, surpassing prior VLA approaches in dexterity, adaptability, and efficiency.

https://github.com/user-attachments/assets/c047bac4-17c3-4d66-8036-badfab2b8c41

🚀 Quick Start: Installation & Deployment

1️⃣ Installation

# Clone the repository
git clone https://github.com/2toinf/X-VLA.git
cd X-VLA

# Create and activate Conda environment
conda create -n XVLA python=3.10 -y
conda activate XVLA

# Install dependencies
pip install -r requirements.txt

or

conda env create -f environment.yml
conda activate xvla-stable

2️⃣ Deploying X-VLA for Inference

X-VLA adopts a Server–Client architecture to separate the model environment from simulation or robot-specific dependencies. This design avoids package conflicts and supports distributed inference across GPUs, SLURM clusters, or edge devices.

🧠 Available Pre-trained Models

• —[ ] We observed a slight performance drop (around 1% across different datasets) after converting our models to the HF format, and we’re actively investigating the cause.

🧠 About Libero Setup and Evluation

• —[x] For questions about converting relative actions to absolute actions and our implementation, please first refer to issue #2 and #15. We have updated full preprocessing guidance here.

🔥 Update: We have released the LoRA fine-tuning code, along with checkpoints and the associated inference code.

🧩 Notes

• —All models share a consistent architecture: configuration_xvla.py, modeling_xvla.py, and unified tokenizer (tokenizer.json).
• —The X-VLA-Pt model is the foundation checkpoint, trained across multiple robot domains.
• —Each embodiment is fine-tuned for its respective environment while retaining cross-embodiment alignment.
• —Evaluation scripts (in evaluation/) follow a standardized format for reproducible benchmarking.

📊 Performance metrics follow standard evaluation protocols detailed in the paper.

3️⃣ Launching the Inference Server

from transformers import AutoModel, AutoProcessor
import json_numpy

# Load model and processor
model = AutoModel.from_pretrained("2toINF/X-VLA-WidowX", trust_remote_code=True)
processor = AutoProcessor.from_pretrained("2toINF/X-VLA-WidowX", trust_remote_code=True)

# Start the inference server
print("🚀 Starting X-VLA inference server...")
model.run(processor, host="0.0.0.0", port=8000)

Once launched, the API endpoint is available at:

POST http://<server_ip>:8000/act

4️⃣ Client Interaction & Action Prediction

The client communicates via HTTP POST, sending multimodal data (vision + language + proprioception) as a JSON payload.

Payload Structure

Example Client Code

import requests
import numpy as np
import json_numpy

server_url = "http://localhost:8000/act"
timeout = 5

# Prepare inputs
proprio = np.zeros(7, dtype=np.float32)
image = np.zeros((256, 256, 3), dtype=np.uint8)
instruction = "Move the gripper to the target position"

payload = {
    "proprio": json_numpy.dumps(proprio),
    "language_instruction": instruction,
    "image0": json_numpy.dumps(image),
    "domain_id": 0,
    "steps": 10
}

try:
    response = requests.post(server_url, json=payload, timeout=timeout)
    response.raise_for_status()
    result = response.json()
    actions = np.array(result["action"], dtype=np.float32)
    print(f"✅ Received {actions.shape[0]} predicted actions.")
except Exception as e:
    print(f"⚠️ Request failed: {e}")
    actions = np.zeros((30, 20), dtype=np.float32)

Expected Output

[Server] Model loaded successfully on cuda:0
[Server] Listening on 0.0.0.0:8000
[Client] Sending observation to server...
✅ Received 30 predicted actions.

5️⃣ Standardized Control Interface: EE6D

To ensure consistency across embodiments, X-VLA adopts a unified EE6D (End-Effector 6D) control space.

⚙️ Reference Post-processing: ``python from datasets.utils import rotate6d_to_xyz action_final = np.concatenate([ action_pred[:3], rotate6d_to_xyz(action_pred[3:9]), np.array([1.0 if action_pred[9] > 0.5 else 0]) ]) `` When feeding proprioception to the model, apply the inverse transformation accordingly.

6️⃣ Reference Client Implementations

Each released model includes a corresponding reference client under `evaluation/<domain>/<robot>/client.py` for reproducing exact deployment behaviors. We strongly recommend adapting from these clients when connecting to physical or simulated robots.

7️⃣ SLURM & Cluster Deployment

For large-scale or distributed training/deployment (e.g., HPC clusters, AgiBot nodes):

python -m deploy --model_path /path/to/your/model

This script automatically detects SLURM environment variables, launches distributed servers, and writes connection metadata to info.json.

⚙️ Training / Fine-tuning on Custom Data

X-VLA supports fine-tuning on new demonstrations via a modular and extensible dataset interface.

Data Preparation Workflow

1. 1.Prepare Meta JSONs — each domain has a meta.json listing trajectory file paths.
2. 2.Implement Custom Handler — write a domain loader class with iter_episode(traj_idx) generator.
3. 3.Register Domain — update:

• —datasets/domain_handler/registry.py
• —datasets/domain_config.py

Example Handlers

Launch Training with Accelerate

accelerate launch \
    --mixed_precision bf16 \
    train.py \
    --models '2toINF/X-VLA-Pt' \
    --train_metas_path /path/to/meta_files.json \
    --learning_rate 1e-4 \
    --learning_coef 0.1 \
    --iters 50000 \
    --freeze_steps 1000 \
    --warmup_steps 2000

📚 Citation

If you use X-VLA in your research, please cite:

@article{zheng2025x,
  title   = {X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model},
  author  = {Zheng, Jinliang and Li, Jianxiong and Wang, Zhihao and Liu, Dongxiu and Kang, Xirui
             and Feng, Yuchun and Zheng, Yinan and Zou, Jiayin and Chen, Yilun and Zeng, Jia and others},
  journal = {arXiv preprint arXiv:2510.10274},
  year    = {2025}
}

🪪 License

This repository is licensed under the Apache License 2.0. You may freely use, modify, and distribute the code under the terms of the license.

Copyright 2025 2toINF (https://github.com/2toinf)
Licensed under the Apache License, Version 2.0.

Maintained by [2toINF](https://github.com/2toinf) 💬 Feedback, issues, and contributions are welcome via GitHub Discussions or Pull Requests.