langzippkkk/bendbench-scoring

Scoring sheet for a counterfactual visual-effects benchmark. Thirty paired clips across three sources; each pair is two generations from one first frame, one letting the scene proceed normally and one replacing exactly one physical rule.

Each card is scored in two stages, and the order matters. Stage 1 is blind: the rater is not told which rule was replaced, and answers in two steps -- first which of the eight law families it belongs to, then which law inside that family. The eight families are the same on every clip, so the vocabulary is learnable rather than thirty unrelated sets of six. Two escape options end stage 1 without a law. It is the only question in the benchmark with a right answer, and revealing the rule first would destroy it. Two frontier vision-language models scored 37% and 53% on the one-step version of this question, and three human raters scored 40%, 50% and 60%; chance on the family step is 12.5%.

Stage 2 reveals the rule, what should be visible in B, and the three declared regions of the frame, then asks four or five binary questions of both halves. Every reading here survived measurement between two judges: the binary probe reached kappa 0.70, against 0.37 / 0.33 / 0.27 for the four-point readings it replaced and 0.13 for a binary question that asked about the whole scene. The dividing line is not how many points a scale has -- it is whether the question names one specific body.

Nineteen of the thirty instances carry a control body: something else in frame governed by the same rule, on which the rule was not replaced. Those get the locus question asked a second time, of that body, where the expected answer is unchanged. The other eleven say so on the card rather than substituting the walls and the furniture.

Access is gated by a shared username and password. Each submission is written as one JSON file into a private dataset repository, so raters never see one another's answers and nothing depends on anyone remembering to export a file.