LCZZZZ/ct-criterion-review

CT / Criterion Timing Review Dataset

A unified dataset of 6,241 proactive-assistant timing samples over 6,215 full-length videos (~128 GB), plus the complete human-review web platform used to audit them.

Each sample pairs a video with a user request (e.g. "Walk me through assembling this side table, and check that I align the leg joints correctly") and a list of help points — the moments where a proactive assistant should speak up, what it should say, and precisely when. The dataset targets the core research question of proactive video assistants: not just what to say, but when to say it.

Composition

The 6,241 samples come from two sub-datasets that share videos and task families but differ in timing granularity:

Task families (balanced by design, 1,553–1,568 samples each):

• —G — real-world guidance: cooking, assembly, repair, barista, DIY (egocentric & tutorial video)
• —L — long-video explanation: lectures, documentaries, activity walkthroughs
• —M — monitoring: surveillance, anomaly/risk watching, counting
• —T — GUI / computer-use tasks: web apps, desktop software, spreadsheets

Videos are full-length (30 s – 600 s, median 124 s) and are drawn from 1,276 sub-sources (agentnet, videocua, fingertip, evostruggle, wearable-AI recordings, HoloAssist, COIN, ActivityNet, MLVU, Video-MME, and many more; the exact sub-source of every sample is in source_canonical).

Help points: 37,174 total, mean 6.0 per sample (1–25). In criterion_timing, every help point is typed with one of 12 criterion types — top ones: next_step_guidance (2,423), knowledge_gap (1,141), progress_summary (1,087), goal_event_notify (893), quality_verification (724), risk_alert (213), error_correction (114).

Repository layout

data/
  unified_review.jsonl     # 6,241 samples, one JSON object per line
  source_snapshot.jsonl    # the raw source rows before unification (verbatim)
  video_uris.txt           # original GCS URIs of the 6,215 videos
media/
  <xx>/<sha256>.mp4        # videos, sharded by first 2 hex chars of sha256
code/
  server/app.py            # review platform backend (FastAPI)
  server/build_dataset.py  # script that produced unified_review.jsonl
  static/index.html        # review platform frontend (single file, no deps)
  run.sh, README.md

Sample schema (data/unified_review.jsonl)

Each element of help_points:

Quick start

import json

samples = [json.loads(l) for l in open("data/unified_review.jsonl")]
s = samples[0]
print(s["question_text"])
for hp in s["help_points"]:
    bt = hp["best_time"]
    t = bt["t"] if bt["type"] == "point" else bt["optimal"]
    print(f"  [{t:7.1f}s] {hp['content'][:80]}")
# the video for this sample:
print(s["video_path"])   # e.g. media/ba/ba3ed0....mp4

With huggingface_hub:

from huggingface_hub import hf_hub_download
path = hf_hub_download("LCZZZZ/ct-criterion-review", s["video_path"], repo_type="dataset")

The review platform (code/)

A self-contained web app for human auditing of exactly this data:

• —sample browser with G/L/M/T color coding, virtual-scrolling list, filtering by family / source / review status, and search
• —video player with a help-point timeline (windows drawn as bands, optimal moments as ticks; click to seek)
• —per help point: a ±10 s @ 1 fps frame strip with click-to-seek and a zoom lightbox (960 px frames, ←/→ stepping), extracted on demand by ffmpeg and cached
• —editable question text, help-point text, and timing (point or start/optimal/end) with validation (non-negative, ordered, ≤ duration)
• —three review verdicts (qualified / modified / rejected), notes, debounced autosave, a global save-all with per-sample failure reporting
• —per-reviewer result files (reviews/<reviewer>/<family>.json) with original + reviewed values, timestamps, and per-sample version numbers for conflict detection (409 on concurrent edits — no silent overwrites)

Run it:

cd code
pip install fastapi uvicorn requests
python server/app.py --port 7867

The platform expects to stream videos over HTTP with Range support; point its video proxy at your local media/ copy or any range-capable store (see server/app.py, /video/ endpoint).

Provenance & license

• —Samples were assembled from a proactive-video benchmark pipeline (ProactiveVideoBench / streaming-timing lines); raw.original_annotation.provenance keeps the full trail for every criterion_timing sample.
• —All annotation content is released under CC-BY-NC-4.0.
• —The underlying videos originate from public research video datasets and are redistributed here for non-commercial research use only. If you are a rights holder and want a video removed, open a discussion on this repo.