adityx23/icl-dataset

ICL Dataset

Bimanual robot manipulation data collected by human teleoperation on the YOR robot, in LeRobot v2.1 format.

3,149 episodes · 2,385,871 frames · 22.1 hours · 36 tasks · 3 camera views

Each episode is a single teleoperated attempt at a manipulation task, recorded at 30 fps with synchronised proprioception, commanded actions and three camera streams.

⚠️ Read this before training on it

This dataset contains failures, trimmed successes and unusable episodes, mixed in with the training set. They are not separated on disk — every episode lives in the same data/ and videos/ directories, distinguished by three boolean flags.

Those four rows are the only combinations that occur. keep implies success and valid, so one test is enough:

from datasets import load_dataset

ds = load_dataset("adityx23/icl-dataset", split="train")

# the balanced training set — 2,134 episodes, 12.5 hours
train = ds.filter(lambda r: r["keep"])

# every real success, balance ignored — 2,715 episodes, 19.2 hours
all_success = ds.filter(lambda r: r["success"])

# everything with something happening in it, successes and failures alike
usable = ds.filter(lambda r: r["valid"])

The flags are also on each record in meta/episodes.jsonl:

train = [e["episode_index"] for e in episodes if e["keep"]]

keep=false successes are good demonstrations — they were cut for balance, not quality. The valid=false ones should not be treated as failure demonstrations: in most of them nothing happens at all, so they are neither a positive nor a useful negative.

filter above works row-wise over frames; group by episode_index (or episode_uid) if you need whole episodes.

Structure

meta/
  info.json             LeRobot v2.1 header (features, shapes, path templates)
  episodes.jsonl        one row per episode: index, task, length, outcome, provenance
  episodes_stats.jsonl  per-episode feature statistics
  tasks.jsonl           36 task instructions, indexed
data/chunk-XXX/episode_NNNNNN.parquet                          tabular, one row per frame
videos/chunk-XXX/observation.images.<cam>/episode_NNNNNN.mp4   h264, 640x480, 30 fps

Chunking is episode_index // 100. Cameras are zed (scene), fish0 (left gripper), fish1 (right gripper).

Per-frame features

Plus the standard bookkeeping columns: timestamp, frame_index, episode_index, index, task_index.

observation.state is measured joint state; everything under action.* is the operator's commanded target. Gripper state is therefore in the action space.

Tasks

Sorted by size of the training set, then by episodes collected.

Balancing

Task sizes were very uneven — one task had 279 successful episodes while others had

1. 1.A model trained on the raw corpus would see one task six times more often than another for no reason other than how collection happened to go.

Successful episodes are therefore capped per task:

The episodes kept are the ones whose duration is closest to that task's median, so what gets held back is the long and short tail rather than an arbitrary slice. Ties break on episode_uid, so the selection is deterministic and reproducible from the metadata alone.

The 581 episodes this excludes keep success=true and valid=true and only get keep=false — they are good demonstrations, cut for balance rather than quality. Nothing was deleted or renumbered: every episode keeps its files, its episode_index and its episode_uid, so the balancing is fully reversible from the keep flag alone.

Two caveats worth knowing:

• —Capping is the only mechanism, so tasks already at or below their cap kept every success they had — including genuine duration outliers. About 1.6% of the balanced set are still 1.5×IQR outliers within their own task.
• —The excluded set is not a random sample. It is the long-duration tail of the six largest tasks, so folding it back in restores volume and the original skew.

Collection

Recorded on a bimanual YOR robot with an NVIDIA Jetson Orin capturing the cameras. An operator teleoperated the robot via VR controllers; the Orin subscribed to the teleop state/action stream and synchronised it with the cameras at 30 Hz.

• —Cameras: one StereoLabs ZED (scene view, left eye) and two Innomaker fisheyes mounted on the grippers, all 640×480 h264.
• —Action convention: end-effector poses are quaternion-first [qw,qx,qy,qz,x,y,z]. When an arm is not under VR control its *_ee target falls back to its measured pose, so the action fields are always well-defined.

Stable episode identity

episode_index is required by LeRobot to run contiguously from 0, so it necessarily changes whenever episodes are added, dropped or reordered — it has shifted meaning in this dataset's history, silently invalidating cached copies.

Every episode therefore also carries `episode_uid`, which does not change across rebuilds:

"episode_uid": "clean_the_plate_20260730_182850:000069"
                └─ collection run ──────────────┘ └ index within that run

It appears in two places: as a field on every record in meta/episodes.jsonl, and as an episode_uid string column in every parquet (constant within a file), so it travels with the frames themselves and is visible in the dataset viewer. That column is an addition to the standard LeRobot v2.1 schema; it is declared in meta/info.json under features, and loaders that select columns explicitly are unaffected.

Use episode_uid as the join key for anything you want to survive a dataset update — annotations, evaluation results, hand-picked subsets. Use episode_index only to locate the parquet and video files in this revision.

The uid is derived from source_dataset and source_episode, so it can be regenerated at any time rather than being an opaque stored value. It changes only if a source collection run is itself renamed or restructured.

Provenance

Merged from 50 separate collection runs. Every episode records where it came from:

Every episode record is five fields — the collection run it came from is the first half of its episode_uid:

{"episode_index": 12, "episode_uid": "2_gear_assembly_20260725_164450:000012", "tasks": ["assemble the two gears"], "length": 2629, "success": true, "valid": true, "keep": false}

run, idx = episode["episode_uid"].rsplit(":", 1)   # -> ("2_gear_assembly_20260725_164450", "000000")

Label quality — please read

The outcome labels are not uniformly reliable, and this is the main limitation.

• —Labels were recorded by the operator at collection time. Some are wrong — most commonly an episode marked success where the robot never moved at all, or where a human hand reset the scene.
• —An automated screening pass flagged 134 episodes as suspected mislabels; a human reviewed all of them and corrected 77. Those carry reviewed_at.
• —The remaining ~3,000 episodes were not individually audited. The screening pass had roughly 34% recall, so additional mislabelled episodes almost certainly remain.
• —Some invalid episodes were explicitly judged unusable by a human; others had no recorded outcome at all and carry merged_from_unknown: true.

If label fidelity matters, filter to episodes carrying reviewed_at for the subset a human has actually confirmed.

Known limitations

• —Failures and unusable episodes are interleaved with successes (see the warning above).
• —Outcome labels are only partially audited.
• —No train/validation split is defined — splits declares everything as train.
• —Camera extrinsics/intrinsics are not published, and the two gripper fisheyes are physically identical models, so left/right assignment relies on the USB port each is plugged into rather than anything recoverable from the files.
• —The three pass X from … to … task pairs differ only in word order, so a bag-of-words text encoder may not distinguish the two directions. Key on task_index if that matters.

Revision history

2026-08-20 (later) — one label string replaced by three booleans. label is gone, split into independent flags: `success` (task achieved), `valid` (usable recording) and `keep` (in the balanced training set). Only four combinations occur, so nothing is lost — keep == true selects exactly the episodes label == "train" did.

Note for anyone on a pre-2026-08-20 copy: `success` changed type, from a string with four values to a boolean. e["success"] == "success" now evaluates to False for every episode instead of raising. Use e["keep"].

2026-08-20 — episode record simplified to five fields. The record had grown to nine fields carrying six facts. source_dataset and source_episode were dropped (both are recoverable from episode_uid, verified 3,149/3,149 in each direction), subset was dropped (superseded by the balancing, empty on 3,049 of 3,149) and balance was dropped (it was non-empty on exactly the excluded episodes, which the label already states).

The outcome field was renamed success -> `label`, with values train / trimmed / fail / invalid replacing success / successful_invalid / fail / invalid. The old vocabulary had a trap: "successful_invalid" starts with "success", so a startswith or substring filter silently returned 2,715 episodes instead of 2,134. No new value is a prefix of another.

This is a breaking change to field names. Counts, episode indices, task indices, videos and frame data are all unchanged — label == "train" selects exactly the episodes success == "success" did.

2026-08-19 — outcome columns added. The balancing previously lived only in meta/episodes.jsonl, which is not part of the default config the dataset viewer and load_dataset() read — so through that path the dataset still appeared unbalanced, with no outcome field at all. success and balance are now columns on every parquet row and declared in meta/info.json. Data, videos, episode indices and task indices are unchanged; only the two columns were added.

2026-08-11 — balanced; `successful_invalid` added. Successful episodes are now capped per task at 100/50/48 (see Balancing), moving 581 episodes from success to the new successful_invalid label. The balanced set is 2,134 episodes / 12.5 hours, down from 2,715 / 19.2. Nothing was deleted, renumbered or re-encoded — only the success field changed, so a cached copy of the data stays valid and the change can be undone from metadata alone. The 105 genuinely invalid and 329 fail episodes are untouched.

2026-08-07 — `episode_uid` and `subset` added. Two string columns were added to meta/episodes.jsonl and to every parquet, and declared in meta/info.json under features (14 features -> 16). episode_uid gives each episode an identity that survives rebuilds; subset marks the trimmed 100-episode selection of put the circle on the peg. Both are additions to the standard LeRobot v2.1 schema — loaders that select columns explicitly are unaffected. No episode index, task index or video changed.

2026-08-07 — task 14 and 16 renamed. Visual review of all 109 episodes in the two cluttered orange-cube tasks showed the cube is placed in the cardboard box, not on the plate — the plate is present as a distractor holder (chilli, eggplant, disc), and in many episodes the cube even starts on it. Both task strings were corrected from "place it on the plate" to "place it in the box". Text only: task_index values, episode indices, parquet contents and videos are all unchanged, so a cached copy stays valid — only the two instruction strings differ. The uncluttered tasks (15, 17) genuinely target the plate and are unchanged.

Note that this makes the four orange-cube tasks not a 2×2 of arm × clutter: cluttered always means the box and uncluttered always means the plate, so the duration difference between them reflects a different task, not the effect of clutter alone.

2026-08-05 — full rebuild. This dataset was rebuilt from source and re-published. It supersedes an earlier version that had different episode indices, different task indices and 44 slug-style task labels ([pnp_left_arm]). Three under-populated tasks were dropped in this rebuild — open the book and scribble in it (5 episodes, 0 successes), stack the six cups into a tower (7 episodes, 1 success) and unscrew the bottle and water the plant (1 episode) — removing 13 episodes.

Any copy downloaded before 2026-08-05 is incompatible: episode and task indices both changed, so a cached copy will not error, it will simply be mislabelled. Re-clone rather than pulling incrementally.

License

Not specified. No licence has been chosen for this data, which means downstream use is legally ambiguous. If you intend to use it, check with the authors first.