mishig/lerobot-camera-frustum

LeRobot Camera Frustum

A demo of `@huggingface/lerobot`: build LeRobot apps without downloading datasets. The package reads any LeRobot dataset on the Hub (v2.0, v2.1 and v3.0) client-side over HTTP range requests, so an app only fetches the metadata, episode index and rows it actually shows. The sidebar counts those bytes against the dataset's full size.

This app replays an SO-100 / SO-101 episode in 3D: the arm (URDF) follows observation.state, and each camera plays as a video on the image plane of its frustum, in sync with the arm.

import { LeRobotDataset } from "@huggingface/lerobot";

const ds = new LeRobotDataset("lerobot/svla_so101_pickplace");
const info = await ds.info(); // fps, cameras, joint names
const [ep] = await ds.episodes({ offset: 3, limit: 1 });
ep.videos; // per camera: url, fromSec, toSec
ep.data; // parquet url, fromRow, toRow

Run locally

npm install
npm run dev

Open http://localhost:5173/?dataset=lerobot/svla_so101_pickplace&episode=0, or type any org/dataset in the sidebar.

Scene

• —Each camera sits at a guessed pose (top, front, side, left and right are inferred from the camera name), with a frustum drawn to its video panel. Datasets don't record camera extrinsics, so the poses are approximate.
• —A camera named wrist / gripper / hand floats next to the gripper and follows it.
• —The arm is driven by observation.state.

Click a panel to fly the camera to it.

Keys

Notes

• —Joint values are converted to radians by range, following the LeRobot dataset visualizer: above 360 means servo ticks, above 2π means degrees, otherwise radians. Datasets recorded in LeRobot's normalized −100…100 range are treated as degrees too, so poses are approximate for them.
• —The SO-101 URDF and meshes come from the lerobot/robot-urdfs Hub bucket; SO-100 datasets are rendered on it as well.
• —Videos stay detached while an episode's rows load. Seeking inside the large shared v3 video files can stall the HTTP/2 connection to the Hub CDN that the parquet and URDF requests also use.
• —The panels are WebGL textures, so the browser has to decode the dataset's codec (most use AV1: recent Chrome, Firefox and Edge; Safari only on hardware with AV1 support).