HaptalAI/robotics-quality-leaderboard

Robotics Dataset Quality Leaderboard

What This Is

This repository hosts an automatically-updated quality leaderboard for robotics imitation-learning datasets on HuggingFace. Each dataset is scored by the HaptalAI quality scorer, an open-source tool that streams a sample of episodes from a dataset, detects the available sensor schema, runs a suite of failure-detection checks, and computes a single 0–100 quality score. The leaderboard is intended as a first-pass screening signal to help practitioners identify datasets worth closer inspection before training — not as a ground-truth data-quality verdict.

Scoring Methodology

The scorer streams the first N = 50 episodes from the train split of each dataset (see Sampling) and runs the checks below. All checks are schema-adaptive: a check is skipped rather than penalised if the required columns are absent. The checks_run and checks_skipped columns in the leaderboard record exactly which checks fired for each dataset.

Physical Anomaly Checks

velocity_spike

Detects episodes containing frames where any velocity component is an outlier at the dataset level.

Procedure:

1. 1.Compute the L2 norm of the velocity vector at each frame: |v|_t = sqrt(sum_i v_i(t)^2)
2. 2.Compute the dataset-wide mean μ and standard deviation σ of |v|.
3. 3.Flag a frame if (|v|_t − μ) / σ > 3.5.
4. 4.Flag an episode if any of its frames are flagged.

Threshold: z-score > 3.5

Column source: Explicit velocity columns (e.g. observation.velocity, joint_velocities) are used when present. When absent, velocity is derived as the per-episode finite difference of state columns: v̂_t = state(t) − state(t−1). Derived kinematics are noisier than measured velocity and produce a higher false-positive rate (approximately 5–10 percentage points higher flag rate on clean data); this is noted in the scorer warnings output and in the Limitations section.

acceleration_spike

Detects sudden changes in velocity indicative of impact, slip, or controller instability.

Procedure: Same as velocity_spike but applied to acceleration magnitude. Acceleration is derived from velocity differences within each episode when explicit acceleration columns are absent.

Threshold: z-score > 4.0 (more permissive than velocity, because legitimate motion naturally produces higher acceleration variance).

torque_saturation

Detects episodes where joint torques or motor efforts reach near-maximum levels, indicating potential hardware stress or aggressive control.

Procedure:

1. 1.Compute |τ| for each frame as the maximum absolute value across all torque columns.
2. 2.Compute τ_max = the global maximum |τ| observed in the sample.
3. 3.Flag a frame if |τ|_t > 0.85 × τ_max.
4. 4.Flag an episode if any of its frames are flagged.

Threshold: |τ| > 85% of the maximum observed torque.

Note: This check uses the observed maximum rather than a nominal hardware limit because nominal limits are not available in dataset metadata and vary across platforms. Using 85% of observed maximum means the check fires when torque reaches near-observed-ceiling for this dataset, not an absolute hardware ceiling. Datasets with uniformly high-torque operation (e.g. heavy manipulation) may show elevated saturation rates without indicating a problem; this is a known limitation.

Column source: Columns matching torque, effort, motor_torque, joint_effort (case-insensitive substring match). Skipped if none found.

force_spike

Detects sudden contact force events — collisions, slips, or unexpected contact.

Procedure: Same z-score approach as velocity_spike, applied to the L2 norm of force/wrench columns.

Threshold: z-score > 4.0

Column source: Columns matching force, wrench, contact, ft_sensor (case-insensitive). Skipped if none found. Most simulated datasets do not include force data; this check is skipped for them.

Misalignment Checks

The scorer uses a two-tier approach. If explicit misalignment columns are present, they are used exclusively. If not, inferred misalignment from action-state correspondence is used.

explicit_misalignment

Used when the dataset includes pre-computed misalignment, residual, or drift columns (e.g. a misalignment_score column written at collection time).

Procedure:

1. 1.Compute the L2 norm of all misalignment columns per frame and normalise to [0, 1] by dividing by the global maximum.
2. 2.Flag a frame if the normalised misalignment > 0.5.
3. 3.Flag an episode if any of its frames are flagged.

Column source: Columns matching misalignment, residual, drift, tracking_error (case-insensitive). When found, action_state_misalignment is automatically skipped to avoid double-counting.

action_state_misalignment

Inferred check for datasets without explicit misalignment columns. Detects episodes where large commanded actions produce negligible state changes — a pattern consistent with lag, unresponsive actuators, or data recording errors.

Procedure:

1. 1.Compute action magnitude per frame: |a|_t = sqrt(sum_i a_i(t)^2).
2. 2.Compute state-change magnitude per frame, within each episode: |Δs|_t = ||s(t) − s(t−1)||.
3. 3.Compute dataset-wide z-scores for both signals: z_a and z_Δs.
4. 4.Mark a frame as suspicious if z_a > 1.0 AND z_Δs < −0.5 (unusually large command with unusually small response).
5. 5.Flag an episode if > 15% of its frames are suspicious.

Thresholds: z(action) > 1.0, z(state_change) < −0.5, per-episode suspicious-frame rate > 15%.

Column source: Requires both action columns and state / observation.state columns. Skipped if either is absent.

repeated_action

Stuck-controller check. Detects episodes where the identical action vector is repeated for many consecutive frames, which is consistent with a frozen controller, a stalled sensor pipeline, or data duplication during recording.

Procedure:

1. 1.Within each episode, compute the row-wise diff of the action matrix.
2. 2.A frame-pair is "identical" if max |Δaction| < 1e-4.
3. 3.Find the longest consecutive run of identical action pairs.
4. 4.Flag an episode if the longest run ≥ 8 consecutive frames.

Threshold: Run ≥ 8 frames with |Δaction| < 1e-4.

Data Integrity Checks

nan_inf

Detects sensor dropouts, recording failures, or corrupt values.

Procedure: Flag any frame containing a NaN or ±Inf in any numeric column. Report both the frame-level rate (fraction of frames affected) and the episode-level rate (fraction of episodes containing at least one bad frame).

constant_sequence

Stuck-sensor check. Detects episodes where a sensor channel that normally varies becomes completely constant — indicative of a dead or frozen sensor, not normal motion.

Procedure:

1. 1.Compute the global standard deviation of each numeric column across all sampled frames. Columns with global std < 1e-3 are excluded — they are expected to be constant (goal positions, fixed setpoints) and would produce 100% false-positive flags.
2. 2.For the remaining "normally variable" columns, compute the per-episode standard deviation.
3. 3.Flag an episode if any variable column has within-episode std < 5% of that column's mean within-episode std across all episodes.

Threshold: Within-episode std < 5% of the column's typical within-episode std.

Score Formula

quality_score = max(0, min(100, round(
    100
    − 75.0 × physical_failure_rate
    − 75.0 × misalignment_suspect_rate
    − 10.0 × data_failure_rate
)))

Where:

Weights rationale: Physical and misalignment penalties are weighted symmetrically at 75 points each, so a dataset with either a ~60% physical failure rate or a ~60% misalignment rate scores near 55. The data-integrity penalty is weighted at 10 points, reflecting that occasional NaN frames are less severe than systematic motion anomalies. The formula was calibrated so that a clean, well-collected dataset scores in the 85–95 range, leaving room for naturally occurring variability (e.g. some episodes being harder than others).

Score bands:

These bands are heuristic thresholds, not certified quality levels. See What a Score Means.

Sampling

The scorer collects the first 50 complete episodes from the train split, streamed in the dataset's native order, up to a hard cap of 100,000 frames. Episodes are delimited by changes in the episode_index column (or equivalent).

Why 50 episodes is a useful screen:

• —For datasets with hundreds of episodes, 50 provides sufficient statistical power to detect systematic issues (e.g. a sensor that is dead in 30% of episodes will appear in ~15 of the 50 sampled ones with high probability).
• —Streaming 50 episodes is practical at inference time without downloading the full dataset, enabling the nightly scoring job to cover many datasets quickly.

Acknowledge the limitation: 50 episodes is a sample, not a full audit. A dataset with rare failure modes in later episodes may score higher than its full-dataset quality warrants. A dataset with a difficult initial task segment may score lower. The num_episodes_sampled column records the exact count for each scored dataset.

What a Score Means

A quality score is a screening signal, not a verdict.

• —A low score means: the sampled episodes contain patterns that match heuristics for physical anomalies, action-state misalignment, or data integrity issues. It means the dataset is worth closer inspection — not that it is unsuitable for training.

• —A high score means: no systematic issues were detected in the heuristic checks on the sampled episodes. It does not mean the dataset is free of problems; it means the scorer's checks did not fire.

• —Scores are relative within similar schema types. A dataset with explicit torque columns and one without are not fully comparable: the former has more checks running, which can only increase the penalty. Use checks_run to understand what was measured.

Limitations

1. 1.Heuristic-based, not ground-truth. Every check in this scorer applies a statistical threshold to a derived signal. These thresholds were chosen to produce low false-positive rates on a reference set of known-clean datasets and known-bad datasets, but they are not validated against human ground-truth annotation at scale.

1. 1.Derived kinematics increase false-positive rate. When a dataset lacks explicit velocity columns, the scorer computes velocity as state(t) − state(t−1). Finite differences of position are sensitive to quantisation, non-uniform timestamps, and configuration-space discontinuities (e.g. angle wrapping). This produces an elevated false-positive rate of approximately 5–10 percentage points on clean data for the velocity_spike check. The scorer warns when derived kinematics are used.

1. 1.Coverage varies by schema. A dataset that exposes velocity, torque, and force columns will have all physical checks run against it. A dataset that exposes only state and action will have at most two physical checks (velocity_spike via derived kinematics, and acceleration_spike). Scores are not fully comparable across datasets with different schemas; inspect checks_run and checks_skipped to understand what the score reflects.

1. 1.Torque saturation uses a relative threshold. The torque check compares against the maximum observed torque in the sample, not a hardware-specific nominal limit. For datasets where all demonstrations are uniformly high-torque (e.g. heavy object manipulation), the saturation rate may be artificially elevated.

1. 1.Sampling order is not random. Episodes are taken in dataset order. If a dataset's first 50 episodes are systematically different from the rest (e.g. calibration trials, easy cases, failed attempts segregated to the beginning), the score may not represent the full dataset.

1. 1.No semantic evaluation. The scorer does not assess task success, reward, or whether the robot achieved its goal. A dataset of consistently failed demonstrations could score high if the trajectories are physically plausible and well-formed.

Reproducibility

The full scorer source is available at: https://github.com/aaravbedi/quality_scorer

Anyone can reproduce any score in this leaderboard by running:

git clone https://github.com/aaravbedi/quality_scorer
cd quality_scorer
pip install -r requirements.txt

python - <<'EOF'
from scorer.scorer import score_dataset
result = score_dataset("lerobot/pusht", n_episodes=50, verbose=True)
print(result["quality_score"])
EOF

Scores may differ slightly across runs due to non-deterministic streaming order on some HuggingFace datasets, and will change if the dataset itself is updated. The scored_at timestamp in the leaderboard records when each score was computed. To reproduce a historical score exactly, use the dataset revision that was current at that timestamp.

Contesting or Improving a Score

Dataset authors who believe their dataset was mis-scored are encouraged to open a discussion on this repository. Common causes of unexpected scores:

• —Non-standard column names the scorer did not recognise (e.g. velocity in a column called qvel instead of joint_velocities). The schema detector uses substring matching against a known vocabulary; edge cases can be added.
• —Non-standard episode delimiters (e.g. no episode_index column, episode boundaries encoded differently).
• —Known data artefacts that are intentional (e.g. a benchmark dataset that deliberately contains failure episodes, which will correctly score low).

To request a schema fix or re-score, open a discussion on this dataset repo, or contact aarav@haptal.ai. Schema fixes are pushed to the scorer and the affected datasets are re-scored in the next nightly run.

Update Cadence

The leaderboard is updated nightly at 02:00 UTC via a HuggingFace Scheduled Job. Each run discovers robotics datasets on HF Hub that are not yet in the leaderboard, scores up to 50 new datasets, and appends results to leaderboard.parquet. Previously scored datasets are not re-scored unless the scorer version changes.

Citation

If you use this leaderboard in research, please cite:

@misc{haptal2025leaderboard,
  title        = {Robotics Dataset Quality Leaderboard},
  author       = {Bedi, Aarav and HaptalAI},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/datasets/HaptalAI/robotics-quality-leaderboard}},
  note         = {Automated quality scoring for robotics imitation-learning datasets.
                  Scorer source: \url{https://github.com/aaravbedi/quality_scorer}}
}

Contact

Aarav Bedi — aarav@haptal.ai Haptal AI — haptal.ai

For schema bugs, score disputes, or dataset submission requests, open a discussion on this repository or email directly.