InternRobotics/VLAC-Cut-Benchmark

1---2pretty_name: Video Progress Benchmark3tags:4  - robotics5  - embodied-ai6  - video-progress7  - progress-estimation8  - reward-modeling9  - benchmark10license: other11---12 13# Video Progress Benchmark14 15<div align="center">16 17[Paper](https://arxiv.org/abs/2607.09776) ·18[Code](https://github.com/InternRobotics/VLAC-cut) ·19[Model](https://huggingface.co/InternRobotics/VLAC-Cut) ·20[Benchmark](https://huggingface.co/datasets/InternRobotics/VLAC-Cut-Benchmark)21 22</div>23 24## Overview25 26The **Video Progress Benchmark (VPB)** evaluates process-level task progress estimation for robot manipulation. It measures whether a model can capture advancement, stagnation, regression, and recovery throughout an execution video.27 28VPB is built from the held-out portion of the Progress Annotation Dataset. Unlike endpoint-only evaluations, VPB focuses on temporal task progress and supports analysis of partial completion, temporary failure, and subsequent recovery.29 30This Hugging Face repository contains the official benchmark splits and video archives. The corresponding preprocessing, inference, and evaluation code is maintained in the GitHub repository.31 32## Dataset Scale33 34| Inventory | Records | Episodes | Tasks | Keyframe progress points |35|---|---:|---:|---:|---:|36| Full annotation inventory | 35,230 | 26,615 | 15,206 | 464,446 |37| Curated train/evaluation inventory | 28,167 | 22,978 | 15,206 | 375,172 |38 39## Official Splits40 41VPB is organized along two axes: whether the semantic task unit appears in the training split, and whether the annotated trajectory contains a regressive progress transition.42 43| Split | Records | Episodes | Keyframe progress points |44|---|---:|---:|---:|45| Train | 24,652 | 20,479 | 331,762 |46| Expert seen | 1,043 | 1,033 | 11,769 |47| Expert unseen | 1,043 | 1,035 | 12,333 |48| Non-expert seen | 713 | 706 | 9,458 |49| Non-expert unseen | 716 | 706 | 9,850 |50 51All views from the same physical execution are assigned to the same split. Held-out progress annotations are excluded from prompt construction, augmentation, in-context demonstration selection, fine-tuning, and checkpoint selection.52 53## Repository Contents54 55```text56splits/57  train.json58  test_expert_seen.json59  test_expert_unseen.json60  test_nonexpert_seen.json61  test_nonexpert_unseen.json62 63data/64  train_videos.tar65  test_videos.tar66 67checksums.sha25668LICENSE69THIRD_PARTY_LICENSES.md70```71 72## Data Format73 74Each split file is a JSON list. Each record includes the trajectory id, task metadata, frame index, dense progress values, semantic anchors, and optional reference context.75 76The split files use the portable frame prefix:77 78```text79__VLAC2_FRAMES_ROOT__/80```81 82Resolve this prefix to the absolute extracted-frame directory when running the GitHub tools.83 84## Evaluation Protocol85 86Annotated semantic keyframes are converted into a canonical reference trajectory by piecewise-linear interpolation. This interpolation is an evaluation convention and does not assume that physical progress changes linearly between events.87 88Global and terminal metrics are computed on the official `1 Hz` evaluation grid. Local direction metrics are computed directly on adjacent annotated semantic anchors. Global metrics are first computed per record and then averaged over metric-valid records so that long videos do not dominate the result.89 90| Scope | Metrics | What they measure |91|---|---|---|92| Global trajectory | MAE, PRC, VOC | Absolute calibration and ordering of progress states; VOC is reported only for expert trajectories |93| Terminal state | TSA, successful F1, failed/incomplete F1, Macro-F1 | Whether the final state is complete using the common `>= 90` threshold |94| Local direction | AP+, AP-, MacroAP | Whether adjacent semantic events are correctly ranked as improvement or regression |95 96Detailed running commands and the complete metric implementation are provided in the GitHub repository:97 98```bash99git clone https://github.com/InternRobotics/VLAC-cut100cd VLAC-cut101 102python scripts/evaluate_vpb_predictions.py \103  --benchmark-root /path/to/VLAC-Cut-Benchmark/splits \104  --predictions predictions/vlac_cut_predictions.jsonl \105  --out-json reports/vlac_cut_vpb_eval.json \106  --out-md reports/vlac_cut_vpb_eval.md107```108 109## Citation110 111Please cite the following paper when using VLAC-Cut, the released model, or the Video Progress Benchmark:112 113```bibtex114@misc{zhai2026helphumanefficientlargescalerobot,115      title={HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation}, 116      author={Shaopeng Zhai and Qi Zhang and Tianyi Zhang and Haoran Zhang and Fuxian Huang and Zhanhui Lin and Zijun Xu and Weinan Zhang},117      year={2026},118      eprint={2607.09776},119      archivePrefix={arXiv},120      primaryClass={cs.RO},121      url={https://arxiv.org/abs/2607.09776}, 122}123```124 125## License126 127The benchmark videos and third-party source data may be subject to additional licenses or terms of use. The source code in the GitHub repository is released under the MIT License.128