HuggingFriends/robojudge_humanstudy

RoboJudge human study — annotation packages

16 self-contained, offline annotation packages covering the 800-item RoboJudge test set.

Together: 800 items, no duplicates, full coverage. All clips belonging to the same (dataset, task, episode) scene stay inside one package, so a scene is never split across two annotators.

How to use

1. 1.Download one zip and unpack it.
2. 2.Double-click index.html — it runs from file://, no server needed.
3. 3.Enter your name, score each clip, then click Save my results to export a JSON file.
4. 4.Send that JSON back to whoever assigned you the package.

Progress is kept in localStorage, so closing the browser does not lose work — same machine, same browser. Opening the package on a different computer or browser starts from zero. An internet connection is required: the videos stream from the public dataset.

What is scored

Per clip, 8 inputs:

• —Physical plausibility (PA), 1–5, with three sub-axes (0 fails / 1 partial / 2 ok): agent integrity · scene & object consistency · interaction realism
• —Instruction alignment (IA), 1–5, with three sub-axes (0 / 1 / 2): right agent acts · right object · goal completed

No free text is collected.

Study design notes

These matter if you analyse the results:

• —Annotators watch the full video at native frame rate. The automatic judges do not — they sample frames (the reference protocol is 4 fps). Annotators therefore see motion artefacts that frame-sampled judges cannot. Human and judge scores are not same-input measurements; treat the human scores as a reference standard rather than as an equal-footing comparison.
• —No ground-truth reference video is shown, and the packages do not contain one.
• —No init frame is shown. The judges receive video + instruction only, so showing an init frame would give annotators strictly more than the judge had.
• —Packages carry only item_id, video_url and instruction — no gold labels and no model predictions, so scoring cannot be anchored by them.