InternRobotics/VLAC-Cut-Benchmark

Video Progress Benchmark

<div align="center">

Paper · Code · Model · Benchmark

</div>

Overview

The Video Progress Benchmark (VPB) evaluates process-level task progress estimation for robot manipulation. It measures whether a model can capture advancement, stagnation, regression, and recovery throughout an execution video.

VPB is built from the held-out portion of the Progress Annotation Dataset. Unlike endpoint-only evaluations, VPB focuses on temporal task progress and supports analysis of partial completion, temporary failure, and subsequent recovery.

This Hugging Face repository contains the official benchmark splits and video archives. The corresponding preprocessing, inference, and evaluation code is maintained in the GitHub repository.

Dataset Scale

Official Splits

VPB is organized along two axes: whether the semantic task unit appears in the training split, and whether the annotated trajectory contains a regressive progress transition.

All views from the same physical execution are assigned to the same split. Held-out progress annotations are excluded from prompt construction, augmentation, in-context demonstration selection, fine-tuning, and checkpoint selection.

Repository Contents

splits/
  train.json
  test_expert_seen.json
  test_expert_unseen.json
  test_nonexpert_seen.json
  test_nonexpert_unseen.json

data/
  train_videos.tar
  test_videos.tar

checksums.sha256
LICENSE
THIRD_PARTY_LICENSES.md

Data Format

Each split file is a JSON list. Each record includes the trajectory id, task metadata, frame index, dense progress values, semantic anchors, and optional reference context.

The split files use the portable frame prefix:

__VLAC2_FRAMES_ROOT__/

Resolve this prefix to the absolute extracted-frame directory when running the GitHub tools.

Evaluation Protocol

Annotated semantic keyframes are converted into a canonical reference trajectory by piecewise-linear interpolation. This interpolation is an evaluation convention and does not assume that physical progress changes linearly between events.

Global and terminal metrics are computed on the official 1 Hz evaluation grid. Local direction metrics are computed directly on adjacent annotated semantic anchors. Global metrics are first computed per record and then averaged over metric-valid records so that long videos do not dominate the result.

Detailed running commands and the complete metric implementation are provided in the GitHub repository:

git clone https://github.com/InternRobotics/VLAC-cut
cd VLAC-cut

python scripts/evaluate_vpb_predictions.py \
  --benchmark-root /path/to/VLAC-Cut-Benchmark/splits \
  --predictions predictions/vlac_cut_predictions.jsonl \
  --out-json reports/vlac_cut_vpb_eval.json \
  --out-md reports/vlac_cut_vpb_eval.md

Citation

Please cite the following paper when using VLAC-Cut, the released model, or the Video Progress Benchmark:

@misc{zhai2026helphumanefficientlargescalerobot,
      title={HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation}, 
      author={Shaopeng Zhai and Qi Zhang and Tianyi Zhang and Haoran Zhang and Fuxian Huang and Zhanhui Lin and Zijun Xu and Weinan Zhang},
      year={2026},
      eprint={2607.09776},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2607.09776}, 
}

License

The benchmark videos and third-party source data may be subject to additional licenses or terms of use. The source code in the GitHub repository is released under the MIT License.