edge-inference/hab3-social-rearrange-fabric

Habitat 3.0 Social Rearrangement — Fabric (Learned Communication)

Trained weights for the Pereason + Go + Fabric multi-agent coordination model on the Social Rearrangement task from Habitat 3.0. Two embodied agents — a Boston Dynamics Spot robot and a humanoid — cooperate to rearrange objects across 37 HSSD scenes.

This model uses Fabric, a learned cross-attention communication module that lets agents exchange compressed multi-modal feature messages each timestep. Fabric fuses semantic (VLM) and geometric (depth) tokens before encoding them into inter-agent messages, enabling richer coordination. Compared to the no-communication Go baseline (69.6% task success), Fabric reaches 76.8% task success with significantly lower collision rates.

This work is part of the thesis "Scalable Multi-Agent Coordination Using a Shared-Context Architecture for Embodied Robotics" by Benjamin Kubwimana. Fabric is the shared-context communication layer explored in the thesis, evaluated here on the Habitat 3.0 social rearrangement benchmark.

What's in this repo

Each checkpoint contains the full model state dict for both agents (keys 0, 1) plus training config.

Task overview

Each episode drops two agents into an HSSD home scene with objects that need to be moved to goal locations. The task is structured as a PDDL planning problem with four subgoal stages:

• —Stage 1.1: Agent 0 (Spot) picks up its target object
• —Stage 1.2: Agent 0 places the object at the goal
• —Stage 2.1: Agent 1 (Humanoid) picks up its target object
• —Stage 2.2: Agent 1 places the object at the goal

Full success (pddl_success) requires both agents to complete all subgoals within 750 timesteps.

Architecture

[image]

Both agents use a hierarchical RL policy with three main components:

Pereason (Perception + Reasoning)

• —VLM backbone: SmolVLM2-500M-Video-Instruct (frozen, 303M params) — processes RGB via a ViT encoder and generates semantic tokens through a truncated language decoder (16 of 32 layers)
• —Depth encoder: Depth Anything V2 Small (trainable, 24.8M params — 22.1M trainable) — encodes depth images into geometric tokens
• —PDDL task instructions are tokenized and fed to the VLM alongside the visual input

Go (Skill Selector)

• —8-block transformer that attends over semantic + geometric tokens via cross-attention, then selects a high-level skill
• —Outputs a categorical distribution over available actions (navtoobj, pick, place, etc.)
• —8 learnable query tokens, 128 context tokens

Fabric (Communication)

• —Geometric projection + pooling: a learned linear layer projects depth tokens (384-dim) to the VLM feature dimension (960-dim), then chunk-based average pooling compresses ~1370 spatial tokens down to 128
• —Token fusion: pooled geometric tokens are concatenated with semantic tokens into a single fused sequence per agent
• —Per-agent MessageEncoder (Linear → GELU → Linear → LayerNorm) compresses the fused multi-modal features into 128-dim messages
• —Per-agent MessageDecoder (cross-attention with gated residual) decodes the partner's message back into the agent's fused feature space; the semantic portion is then extracted for downstream use
• —Gate bias initialized at -2.0 so communication starts near-zero and ramps up as training progresses
• —Symmetric t-1 communication: both agents read from the previous timestep's bus, eliminating the asymmetry where agent1 sees current features but agent0 only sees stale ones
• —Original full-resolution geometric tokens bypass Fabric and go directly to Go alongside the enriched semantics

Trainable parameters (per agent)

Low-level skills

Oracle navigation + learned manipulation skills (pick, place, navtoobj, etc.) using privileged simulator information.

Observations per agent:

• —RGB camera image (arm cam for Spot, head cam for Humanoid)
• —Depth camera image
• —Binary is_holding flag
• —GPS+compass to object start/goal positions
• —Relative GPS to the other agent

Training details

Results

Comparison with no-communication baseline

Training curve

How to evaluate

Requires Habitat 3.0 (v0.3.3) with habitat-baselines, habitat-sim, and the Pereason+Go+Fabric policy classes.

python -u -m habitat_baselines.run \
    --config-name=social_rearrange/pereason_go_fabric \
    habitat_baselines.evaluate=True \
    habitat_baselines.eval.should_load_ckpt=True \
    habitat_baselines.eval_ckpt_path_dir=model-latest.pth \
    habitat_baselines.test_episode_count=50 \
    habitat_baselines.num_environments=1 \
    habitat.dataset.data_path=data/datasets/hab3_episodes/val/social_rearrange.json.gz \
    habitat.dataset.scenes_dir=data/scene_datasets \
    'habitat_baselines.eval.video_option=["disk"]'

Dependencies

• —SmolVLM2-500M-Video-Instruct (downloaded automatically)
• —Depth-Anything-V2-Small-hf (downloaded automatically)
• —Habitat 3.0 with HSSD scenes and hab3_episodes dataset

Citation

@mastersthesis{kubwimana2026scalable,
  title  = {Scalable Multi-Agent Coordination Using a Shared-Context Architecture for Embodied Robotics},
  author = {Kubwimana, Benjamin},
  year   = {2026},
  school = {Georgia Institute of Technology},
  note   = {Model weights: \url{https://huggingface.co/edge-inference/hab3-social-rearrange-fabric}}
}

Built on the Habitat 3.0 platform:

@inproceedings{puig2023habitat3,
  title     = {Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots},
  author    = {Puig, Xavier and Undersander, Eric and Szot, Andrew and Cote, Mikael Dallaire and Batra, Dhruv and Berges, Vincent-Pierre and others},
  booktitle = {ICLR},
  year      = {2024}
}

License

MIT. The underlying Habitat platform and HSSD scenes have their own licenses — see the Habitat 3.0 repo for details.