itspublu/EgoSieve-Eval

EgoSieve-Eval

EgoSieve-Eval is the metadata-only, source-grouped evidence index used for EgoSieve-S v0.1. It contains 1407 labeled window rows across 992 train, 219 validation, and 196 test examples. Source and generated videos are deliberately not redistributed.

What the labels mean

Readiness and boundary targets are derived from HoloAssist v1_1 fine-action intervals using a published fixed-grid occupancy rule. The test set contains 0 direct-human and 142 human-derived readiness rows. Human-derived rows are not independent judgments under the EgoSieve rubric; the recorded review count applies to HoloAssist source interval review.

low_hand_activity is an occupancy proxy. acting_hand_not_visible follows HoloAssist's acting-hand visibility modifier; it does not assert that every hand is absent from a frame. Blur, exposure, camera instability, duplicate frames, and scene-cut labels are paired controlled corruptions versus unmodified references. Their metrics measure that discrimination setting, not human-audited natural prevalence.

Held-out issue rows by provenance: 0 human, 109 human-derived, and 36 controlled-corruption. Unknown task targets remain null and masked.

Splits and leakage

Splits are assigned by original HoloAssist source video, so a source window, its unmodified reference, and every generated variant stay in one split. The exact membership is in evidence/splits.json; raw held-out predictions and recomputed metrics are included alongside it.

Media, license, and privacy

This repository contains labels, source identifiers, timestamps, and model outputs only. Obtain source media from the HoloAssist project under its terms. HoloAssist declares CDLA-Permissive-2.0. First-person video can contain faces, screens, homes, and bystanders; the absence of media here is intentional and does not replace the source dataset's consent and privacy documentation.

The transport mirror used for the local build was pinned for byte stability but did not declare its own license or establish publisher byte equivalence; it is not presented here as the rights source.

Reproducibility and limitations

The build recipe is in `scripts/build_v01_corpus.py`. Every label row carries task-level validity and provenance. These weak-label metrics should not be read as performance on a direct, independently annotated in-the-wild benchmark, and the model must not be used for robot control.

Citation

@inproceedings{wang2023holoassist,
  title={HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World},
  author={Wang, et al.},
  booktitle={ICCV},
  year={2023}
}