Datasetpublic

mulligan/real-routing-d2-r00-r05-eval

Route Cable: round 0-5 evaluation Real-robot held-out evaluation for the Mulligan paper, one dataset per task-round. Episodes from every evaluation session of this round are joined on the initial state (meta/start_table.parquet: one row per start, one column group per policy). Sessions session_id is the fixed release block ID of the recording; it never gets renumbered. Session Results saved Sobol seed Start indices Cameras Original recording b01… See the full description on the dataset page: https://huggingface.co/datasets/mulligan/real-routing-d2-r00-r05-eval.

sourceHugging Faceapache-2.0updated 3d agoView on Hugging Face

0likes932downloads

Dataset Card

Route Cable: round 0-5 evaluation

Real-robot held-out evaluation for the Mulligan paper, one dataset per task-round. Episodes from every evaluation session of this round are joined on the initial state (meta/start_table.parquet: one row per start, one column group per policy).

Sessions

session_id is the fixed release block ID of the recording; it never gets renumbered.

Session	Results saved	Sobol seed	Start indices	Cameras	Original recording
b01	2026-09-09	2026090701	0-49	4	real01b-routing-d1-umirel-lineage-15arm-heldout-sobol50-s2026090701

Policies

Session	Policy	Method	Role	Successes
b01	`baseline_umirel_r0_final`	HG-DAgger	counted	1/50
b01	`ours_umirel_r0_final`	HG-DAgger+Mulligan	counted	3/50
b01	`baseline_umirel_r2_final`	HG-DAgger	counted	4/50
b01	`ours_umirel_r2_final`	HG-DAgger+Mulligan	counted	3/50
b01	`baseline_umirel_r4_final`	HG-DAgger	counted	7/50
b01	`ours_umirel_r4_final`	HG-DAgger+Mulligan	counted	5/50
b01	`baseline_umirel_r6_final`	HG-DAgger	counted	8/50
b01	`ours_umirel_r6_final`	HG-DAgger+Mulligan	counted	11/50
b01	`ours_umirel_r6_iql_g0997_n32`	HiL-IDQL+Mulligan	counted	10/50
b01	`baseline_umirel_r8_final`	HG-DAgger	counted	5/50
b01	`ours_umirel_r8_final`	HG-DAgger+Mulligan	counted	16/50
b01	`ours_umirel_r8_iql_g0997_n32`	HiL-IDQL+Mulligan	counted	17/50
b01	`baseline_umirel_r9_final`	HG-DAgger	counted	10/50
b01	`ours_umirel_r9_final`	HG-DAgger+Mulligan	counted	14/50
b01	`ours_umirel_r9_iql_g0997_n32`	HiL-IDQL+Mulligan	counted	17/50

Pooled by checkpoint (success rate over every eligible block in this round):

Policy key	Successes	Sessions
`hg_dagger_r0`	1/50	b01
`hg_dagger_r1`	4/50	b01
`hg_dagger_r2`	7/50	b01
`hg_dagger_r3`	8/50	b01
`hg_dagger_r4`	5/50	b01
`hg_dagger_r5`	10/50	b01
`hg_dagger_mulligan_r0`	3/50	b01
`hg_dagger_mulligan_r1`	3/50	b01
`hg_dagger_mulligan_r2`	5/50	b01
`hg_dagger_mulligan_r3`	11/50	b01
`hg_dagger_mulligan_r4`	16/50	b01
`hg_dagger_mulligan_r5`	14/50	b01
`hil_idql_mulligan_r3`	10/50	b01
`hil_idql_mulligan_r4`	17/50	b01
`hil_idql_mulligan_r5`	17/50	b01

role is counted for the paper's headline methods and no-cf-ablation for the HG-DAgger+Mulligan actor trained without counterfactual demonstrations (paper CF ablation).

Pooling rule

1.Pool by checkpoint (actor weights + critic + N), not by method name.
2.Eligible blocks are the round's final evaluations only: blinded, interleaved, on fresh Sobol starts disjoint from training. Critic-selection screens and superseded blocks are excluded.
3.Success rates pool every eligible block the checkpoint ran on in that round.
4.Paired tests pair only within a session, stratified by session. Never pair across sessions.
5.A repeat of a whole evaluation block on the same starts replaces the earlier block. Drift checks are excluded.
6.The plotted (deployed) point in each round is the policy that collected the next round's data.

Excluded episodes

None.

Cameras and fields

Cameras: observation.images.wrist_left, observation.images.wrist_right, observation.images.side_1, observation.images.side_2. Where sessions recorded different camera sets, the dataset keeps the cameras every session shares; missing streams are never synthesized.

Videos are the original recordings (byte-identical files, never re-encoded). Outcomes and step counts in meta/episode_provenance.parquet are the reviewed values used by the paper and the Policy Arena; per-session source results, manifests and review records are preserved under meta/sessions/.

Files

—meta/episode_provenance.parquet: session, original repository/revision/episode, start key, policy, model IDs, role, outcome.
—meta/start_table.parquet: per-start outcomes by policy.
—meta/round_dataset.json: sessions, policies, pooled counts and the lock hash.
—meta/source_dataset_lineage.json: the original recordings this dataset was built from.

Release: release-2026-09-24-round-datasets. Paper: real-world results table and appendix "Evaluation protocol".