petkopetkov/longhorizon-orchestrator-benchmark

LongHorizon Orchestrator Benchmark

An offline benchmark for the judgements a manipulation orchestrator delegates to a vision-language model. An orchestrator wraps a frozen low-level policy and replaces a compound instruction ("put everything in the bin") with a stream of single-object subtasks; to do so it must plan (decompose the instruction into subtasks), verify (judge from pixels whether the current subtask is finished), track state (know which goals are already done), and — in its stepwise form — name the next subtask directly from the current frame. This dataset isolates each of those judgements so VLMs and orchestrator configurations can be ranked in minutes of API calls instead of hours of simulator rollouts. The top of the offline ranking is what earns a full closed-loop evaluation.

Everything is derived from three simulated long-horizon manipulation suites (SimplerEnv-Comp, WidowX; RobotArena-Comp, WidowX; RoboLab-Comp, Franka) and labeled by the simulator's own privileged per-goal predicate — the same signal that scores the closed-loop evals, so offline numbers and simulator numbers share one definition of success.

Configurations

The two configurations share task identity (benchmark, scene) and the reference plan (subtasks), which is what lets a scorer cross-check one against the other (see Staying in sync, below).

decomposition (143 tasks)

One row per task: the reset frame a deployed planner would see, the compound instruction, and the reference decomposition that defines success for that task.

subtask_states (552 samples, ≤4 per task)

Mid-episode frames cut from recorded policy rollouts of the same suites, each carrying the full privileged goal state at that instant. One sample answers all three frame-level questions.

Sampling rules that make a label trustworthy: samples are cut only where the goal vector is stable for ±8 steps (the predicate settles a few steps after a release and can flicker while an object rolls); in-progress negatives are cut 24 steps before a completion, clear of the ambiguous hover phase; recordings whose frame count disagrees with the logged step count are refused whole. Frames come from real rollouts of several orchestrator variants (flat, oracle, plan_periodic, plan_gripper, plan_verifier_local) and both successful and failing episodes, so the state distribution is the one a deployed verifier actually faces.

Per task the dataset keeps at most four samples — positives, in-progress negatives, and regression frames where the task produced them — so a full sweep of every question over every sample is a few hundred API calls. Ten further tasks (five per SimplerEnv/Comp and RobotArena/Comp side) have no rollout yet that reaches them; each carries two cold samples cut from its own reset frame (all goals undone by construction), tagged cold with arm: reset_frame, until rollouts arrive.

The four benchmark tasks

Goal matching for decomposition and next_subtask is on (object, destination) head nouns with multiplicity, so wording differences don't count as errors; a genuine synonym still scores as wrong, which makes naming-error counts an upper bound. redundant deserves emphasis: it is the bookkeeping failure an orchestrator pays real budget for (re-commanding finished work).

What is deliberately not covered: the trigger, withholding, revocation, and retry/defer policies. Those are deterministic code whose value depends on closed-loop dynamics (a verifier is queried on states its own decisions induced), so they remain simulator ablations. This benchmark ranks the models behind a variant; it does not replace the closed-loop comparison of scheduling rules.

How faithful is it to a real simulator eval?

The design rule is drive the shipped component, not a restatement. The reference scorer runs the exact Decomposer, Verifier, and stepwise subtask generator classes the closed-loop evals run, with their deployed prompts (not overridable — prompt tuning against this benchmark measured as pure lexical leakage) and their deployed reply parsing. Sample frames are taken from real eval recordings at deployment-relevant moments, and labels are the environment's own success predicate.

Known differences from the closed loop, stated so nobody discovers them the hard way:

• —next_subtask calls are history-free (each sample stands alone); the deployed stepwise generator also sees its own previous commands.
• —The verifier's delta-frame variant (previous query frame as occlusion evidence) is not benchmarkable in v1 — samples carry no previous-query frame.
• —Labels inherit the predicate's semantics: an object held at the goal surface counts as placed, and shared targets check no contact force. Numbers are agreement with the environment, not with a human.

Current coverage (v2, 2026-09-11)

decomposition is complete: all 143 tasks. subtask_states covers all 143 tasks (552 samples, 41.7% positive):

Sample tags: fresh 229, in_progress 166, regressed 80, cold 69, terminal 8. Producing arms: oracle/oraclelatched, planverifier_local, tiered/capx/tiptop reference policies.

Baseline results

v2 baselines on the current revision are running and replace this section when done. Below are the superseded v1 numbers (2026-08-20/21, 108-task / 171-sample revision, pre-unification prompts) for archaeology — do not compare them against v2: the tasks, the prompts, and the sample set all changed.

verification (171 samples):

The models fail differently, which is the discrimination this benchmark exists to expose: 3.1-pro misses almost no completion but calls a third of the undone goals done; gemma-3-4b degenerates into answering "done" to nearly everything (recall 1.00 at precision 0.32). False positives on regressed frames are the expensive kind -- a verifier that cannot see a goal came undone never triggers repair.

state_tracking and next_subtask (same models):

Note the dissociation: gemma-3-4b is catastrophic at verification yet beats Qwen3-VL-2B at generating the next subtask -- judging completion and proposing work are different capabilities, and an orchestrator can mix models per role. The same dissociation shows at the top: gemini-3.6-flash is only mid-field at verification yet generates the most valid next subtasks of any model tested (151/171), at a fraction of 3.1-pro's cost and latency.

decomposition (v1: full then-current 108 tasks, deployed prompt, --max-tokens 8192, temperature 0):

goals_invented > 1 is degeneration — a planner repeating lines until the token cap (a Qwen trait below 32B; count degenerate scenes before comparing means). An earlier six-planner run on the 2026-08-05 92-task revision, for the record: exact plans of 92 — Qwen3-VL-2B 49, Gemma-3-4B 51, Qwen3-VL-8B 55, Qwen3-VL-32B 57, Gemini 3 Flash 76, Gemini 3.1 Pro 74. The scoring rules and caveats live with the reference scorer.

Does the offline score predict closed-loop outcomes?

Measured directly on the v1 revision (2026-08-20/21): the same six verifier models were dropped into otherwise identical closed-loop SimplerEnv-Comp evals — cached plans, same frozen policy, matched 12 units × 24 episodes per arm (288) — bracketed by a no-VLM floor (gripper-open heuristic) and a privileged-state oracle.

Spearman(offline verification accuracy, closed-loop success) = 0.89 across the six verifier arms, and the in-loop precision/recall preserve each model's offline failure signature (gemma's says-done-to-everything profile: offline R 1.00 / P 0.32 → in-loop R 0.98 / P 0.49). Three honest limits: the middle of the ranking compresses (Qwen 8B and 2B tie in the loop, and the two mid-field Gemini models swap within noise — 8B's recall halves on mid-attempt stream frames the offline flip-adjacent samples under-represent); the absolute spread is small because verification is not this policy's binding constraint (the privileged oracle itself only reaches 0.490 — within noise of the best VLM verifier); and per-arm SE is ≈0.03, so neighbouring rows are not individually separated. Use the offline score to rank and screen orchestrator VLMs — the top offline model was the top closed-loop model — not to forecast the size of the closed-loop gain, which depends on the policy.

Usage

With the datasets library:

from datasets import load_dataset

decomposition = load_dataset("petkopetkov/longhorizon-orchestrator-benchmark", "decomposition", split="train")
samples = load_dataset("petkopetkov/longhorizon-orchestrator-benchmark", "subtask_states", split="train")

sample = samples[0]
question = sample["subtasks"][sample["query_idx"]]   # e.g. "put the banana in the bin"
truth = sample["label"]                             # simulator ground truth for that subtask
frame = sample["image_main"]                        # PIL image at that instant

A minimal verification eval with any chat-vision model (the deployed prompt — results are comparable across models only under it):

import json, re

PROMPT = (
    "You are checking the CURRENT state of a robot tabletop scene from its camera views. The subtask "
    "below moves ONE object to a goal location, and the scene may hold SEVERAL interchangeable "
    "objects of that same kind. Count how many DISTINCT objects of the kind the subtask names are in "
    "the subtask's END STATE RIGHT NOW -- at the goal location the subtask asks for, regardless of "
    "how they got there. An object counts as placed when any part of it rests on the goal location, "
    "INCLUDING when partially hidden by the robot arm or other objects. An object the gripper is "
    "still holding does NOT count, however close to the goal location it is held: the release is not "
    "yet a placement and it is the release that most often fails. An object away from the goal "
    "location, or lifted well above it, is not counted. Count each "
    "physical object at most once, however many views show it. First write one short line listing "
    'what you counted, then output STRICT JSON: {"count": <integer>}.'
)

correct = 0
for sample in samples:
    images = [img for img in (sample["image_main"], sample["image_wrist"]) if img is not None]
    subtask = sample["subtasks"][sample["query_idx"]]
    required = sample["subtasks"][: sample["query_idx"] + 1].count(subtask)
    reply = my_vlm(PROMPT, f"Subtask: {subtask}", images)
    verdict = json.loads(re.search(r"\{.*\}", reply, re.DOTALL).group(0))["count"] >= required
    correct += verdict == sample["label"]
print(f"verification accuracy: {correct}/{len(samples)}")

For state tracking, ask the same question once per distinct subtask string; a duplicated subtask ("put the carrot on the plate" ×3) counts as done only when all of its instances are, since a yes/no on the text cannot single out one instance. For next_subtask, send the compound instruction instead and check the reply against the unachieved entries of zip(subtasks, achieved) — exclude samples where all(achieved).

Provenance and staying in sync

The decomposition rows are rendered by each suite's export_scenes tool directly from its task registry; the subtask_states rows are cut from recorded eval episodes and validated, at harvest time, against those same manifests. The harvester, reference scorer and tools live in-tree at barrel/components/inference/api/client/metrics/tools/{export,score}_subtask_states.py and score_decomposition.py (2026-09-11 revision, commit be4a8729f plus the orchestrator-benchmark fixes). The reference scorer re-validates every sample against the decomposition config at load time and refuses to score on drift. So when a task changes: re-export its suite's scenes, republish decomposition, and every stale sample fails loudly until re-harvested from post-change recordings. Rows carry provenance (arm, source, step) so any label can be traced to the exact recording frame it came from.

Rules

• —No privileged input: a scorer gets the frames and the text a robot would have — never the scene's object list.
• —No prompt tuning against these tasks: measured on the decomposition task, prompt rewrites gained nothing once the dataset's object nouns were removed; report deployed-prompt numbers.
• —Don't starve reasoning models of tokens: hidden thinking counts against max_tokens, and a truncated reply is indistinguishable from a wrong one. Decomposition uses 8192 by default.