Voxel51/tavis-head-reachy2-800ep

This dataset was made available by FiftyOne.

Installation

If you haven't already, install FiftyOne:

pip install -U fiftyone

Usage

import fiftyone as fo
from fiftyone.utils.huggingface import load_from_hub

# Load the dataset
# Note: other available arguments include 'max_samples', etc
dataset = load_from_hub("Voxel51/tavis-head-reachy2-800ep")

# Launch the App
session = fo.launch_app(dataset)

Dataset Card for TAVIS-HEAD Reachy2 (800-episode FiftyOne dataset)

[image]

This is a FiftyOne-formatted, self-contained LeRobot v3.0 export of the full tavis-benchmark/tavis-head-reachy2 dataset (all 800 episodes) loaded via fiftyone.utils.huggingface.load_from_hub.

Dataset Details

Dataset Description

TAVIS-HEAD/Reachy2 is one of four multi-task suites in TAVIS (A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning). It contains 800 VR-teleoperated demonstrations of a simulated Pollen Reachy 2 humanoid torso performing five active-vision manipulation tasks (global visual search and clutter handling) in NVIDIA IsaacLab. Episodes record a 19-D bimanual+neck action space, full proprioceptive state, end-effector poses, and four synchronized RGB video streams (head, two wrists, fixed external). This card documents the FiftyOne re-export of the complete source dataset; no episodes were excluded.

• —Curated by: the TAVIS benchmark authors (org: tavis-benchmark on Hugging Face; paper under double-blind review, authors not yet disclosed)
• —Shared by: tavis-benchmark; this FiftyOne re-export shared by the FiftyOne community
• —Language(s): English (33 distinct language_instruction values across the source dataset; 5 task classes on this Reachy2 shard)
• —License: CC-BY-4.0 (datasets); code is MIT. Robot models, YCB objects, and task assets carry their own upstream licenses (NVIDIA IsaacLab, the YCB project, and the original robot-model authors) — see the tavis-benchmark org card

Dataset Sources

• —Repository: tavis-benchmark/tavis-head-reachy2
• —Org / benchmark overview: tavis-benchmark
• —Code (anonymous mirror, review period): https://anonymous.4open.science/r/tavis-F5D7
• —Paper: Under double-blind review at time of this export. Cite by name: "TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning" — BibTeX to be posted once review concludes.
• —Pretrained baselines: tavis-benchmark/pi0-tavis-head-reachy2-headcam, tavis-benchmark/pi0-tavis-head-reachy2-fixedcam (π₀ policies trained on this suite)

Uses

Direct Use

• —Training and evaluating imitation-learning policies (diffusion policy, π₀, etc.) on active-vision, humanoid manipulation tasks.
• —Studying anticipatory gaze behavior via the GALT (Gaze-Action Lead Time) metric, computed from the 19-D commanded-action trajectory (neck + arm channels) — see docs/galt.md in the code mirror.
• —Comparing head-mounted vs. fixed-camera policies (paired π₀ checkpoints are released for both camera modes).
• —Browsing, filtering, and re-exporting episode subsets with FiftyOne before training.

Out-of-Scope Use

• —This is simulated data (NVIDIA IsaacLab); it is not validated for direct sim-to-real transfer without further work.
• —Not intended as a general humanoid-manipulation dataset outside the five TAVIS-HEAD task definitions below.

Dataset Structure

This dataset is a multimodal FiftyOne dataset. Each sample is one episode; the underlying frame-level Parquet rows and MP4 videos are referenced (not duplicated) through media_reference and are exposed in the FiftyOne App's State & Action / Streams tabs.

Fields

Per-frame features (referenced via media_reference, from meta/info.json)

Label types and why

task is a StringField, not a Classification — the source stores task identity as a per-episode string keyed against meta/tasks.parquet (5 distinct task classes: ClutterPickCubeTask, ClutterPickLiftTask, ConditionalPickTask, MultiShelfScanTask, WaitThenActTask), and every episode in this shard has exactly one task, so it is imported as free text on the sample rather than a per-frame primitive. language_instruction (the natural-language variant, e.g. "Find the red cube and pick it up.") remains inside the referenced per-frame Parquet data, not on the sample.

dataset.info contents

dataset.info["lerobot"] records the source codebase_version (v3.0), the original chunks_size / file-size targets, and skipped_episodes ([] — every episode from the source imported cleanly).

Parsing decisions

• —Full dataset, no subsetting. All 800 episodes and all 4 camera streams were imported; the source dataset's single data shard (data/chunk-000/file-000.parquet) and 30 video shard files (4 cameras × up to 9 files each) were downloaded in full (~4.3 GB), so no shard-0-only subset was needed.
• —`meta/tasks.parquet` repair. The source file stored task names as the pandas index (__index_level_0__) rather than a task column, which FiftyOne's importer tolerates but the LeRobot export path (lerobot_export._read_source_tasks) requires as literal task_index + task columns. Repaired locally with reset_index().rename(columns={"index": "task"}) before ingest; no data values were changed.
• —Stale `data/file_index` in `meta/episodes` repair. The source episode metadata table declared data/file_index values 0–4 (as if the frame data were split across five Parquet shards), but the repo ships only one data file (data/chunk-000/file-000.parquet), which was verified to contain all 231,286 frames across all 800 episodes. This is a source metadata bug (the data was likely consolidated into one file after the per-episode index was written but the index wasn't refreshed). Repaired locally by setting data/file_index = 0 for every row before ingest/export; no frame data was moved or altered. Video shard indices (videos/<key>/file_index, genuinely split across up to 9 files per camera) were left untouched.
• —Re-export. Pushing via dataset_type=fo.types.LeRobotDataset re-indexes episode_index to a contiguous 0..799, remaps task_index, and recomputes per-episode and global stats over the full 800-episode set — all of which match the source since no episodes were dropped.
• —`robot_type: null`. The source meta/info.json declares "robot_type": null despite the dataset name/org page identifying the robot as a simulated Pollen Reachy 2; this is a source metadata gap, not a FiftyOne import artifact, and is preserved as-is.
• —Codec caveat. Video is AV1/yuv420p, which decodes fine in Chromium-based browsers (the FiftyOne App); other browsers may have weaker AV1 support.
• —Excluded modalities: none. Every feature declared in meta/info.json is either a supported video stream or a scalar/vector field carried in the referenced Parquet data.

Dataset Creation

Curation Rationale

TAVIS-HEAD isolates global visual-search behavior (as opposed to TAVIS-HANDS, which tests local/occlusion-driven perception) by requiring the robot to scan clutter, shelves, or cue cards with its head before acting. The Reachy2 shard pairs with a matching GR1T2 shard so the benchmark can compare embodiments on an identical task and action-space definition.

Source Data

Data Collection and Processing

Demonstrations were collected via VR teleoperation (Meta Quest) in NVIDIA IsaacLab simulation, built on the LeRobot framework and IsaacLab-Arena. Robot models are from the upstream Pollen Reachy 2 USD distribution; manipulated objects are from the YCB project. Episodes were exported as LeRobot v3.0 (Parquet action/state tables + AV1 video shards, chunked by file size).

Who are the source data producers?

VR teleoperators recorded the demonstrations under the TAVIS benchmark project (org tavis-benchmark); individual operator identities are not disclosed in the source metadata.

Annotations

No separate annotation pass — task identity and language instructions are recorded directly during teleoperation as task_index/task and per-frame language_instruction fields.

Personal and Sensitive Information

None. All data is synthetic (simulated humanoid, simulated objects); no real people, locations, or personal data are involved.

Citation

Until the formal citation is posted, cite by name: "TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning."

APA:

TAVIS Benchmark Authors. (Under review). TAVIS: A Benchmark for Egocentric Active Vision and Anticipatory Gaze in Imitation Learning [Data set]. Hugging Face. https://huggingface.co/datasets/tavis-benchmark/tavis-head-reachy2

More Information

Part of the four-suite TAVIS benchmark (tavis-head-gr1t2, tavis-head-reachy2, tavis-hands-gr1t2, tavis-hands-reachy2; 2,200 total demonstrations). See the tavis-benchmark org page for the full task list, evaluation protocol (id / ood_spatial / ood_init_pose), and pretrained π₀ checkpoints.

Dataset Card Authors

Harpreet Sahota

Dataset Card Contact

Harpreet Sahota