VishalN17/ar-scene-graph-env

AR Scene Graph Builder — OpenEnv

 

An advanced Reinforcement Learning (RL) environment for AI agents to master AR spatial placement. This environment simulates the complex decision-making logic of Meta's Ray-Ban Meta and Orion AR glasses, specifically how they anchor virtual content (notifications, UI panels, navigation) into real-world physical spaces.

🏗️ System Architecture

graph TD
    A[RL Agent / LLM] -->|Action| B[FastAPI Server]
    B -->|Schema Validation| C[ARSceneGraphEnv]
    C -->|State Update| D[NumPy Geometry Engine]
    D -->|Coordinate Check| E[Surface/Obstacle Map]
    C -->|Reward Logic| F[Scoring Engine]
    F -->|Baseline Bonus| G[Anchor Appropriateness]
    F -->|Dimming Penalty| H[Lighting Model]
    C -->|Observation| A

🚀 Key Features

1. High-Fidelity AR Anchoring

The environment introduces critical spatial computing concepts for agent training:

• —`world_locked`: Fixed to global coordinates (best for static furniture/screens).
• —`head_locked`: Fixed to the user's viewport (best for HUD/ambient alerts).
• —`gaze_anchored`: Floats in view with smooth inertia/drift (best for notifications).

2. Environment Diversity

Scenario-based testing across multiple high-context environments:

• —🏠 Indoor Room: Controlled baseline.
• —🏢 Office / Restaurant: Dense obstacle navigation.
• —🏙️ Outdoor Street: Wide-area navigation with dynamic safety risks.
• —📦 Warehouse / Retail: Multi-user spatial coordination.

3. Dynamic Physics & Visibility

• —Lighting Models: bright, dim, outdoor_day, outdoor_night. Reduced visibility scales the gaze-alignment reward, forcing agents to find better-lit placement zones.
• —Dynamic Elements: AI-controlled pedestrians and vehicles that agents must detect and avoid to maintain safety scores.

📈 Baseline Performance

Evaluated on Qwen2.5-72B-Instruct via Hugging Face Router.

🛠️ Installation & Usage

1. Local Setup

pip install -r requirements.txt
python -m uvicorn api.server:app --port 7860

2. Run Inference Loop

# Set HF token in .env or shell
python inference.py

3. API Sample

Place a virtual_screen on a wall:

POST /step
{
  "action_type": "place",
  "position": [2.5, 1.5, 3.0],
  "anchor_type": "world_locked",
  "anchor_surface_id": "wall_n"
}

⚖️ Reward Function

The environment provides a dense, shaped reward in the range [0, 1]:

$$R = \sum (wi \cdot si) - \text{Penalties} + \text{AnchorBonus}$$

• —AnchorBonus: +0.10 for stable anchoring (e.g. Head-locked for Notifications).
• —Visibility Scaler: Gaze alignment is multiplied by the environment light factor.
• —Comfort Zone: Objects must lie within the user's 45° ergonomic cone.

📝 Compliance

• —OpenEnv Compatible: Fully passes openenv validate.
• —FastAPI Standard: Native Pydantic v2 validation.
• —Licensing: Open-source MIT.