neurips2026-anonymous/signalbench-alfworld-bkp

SignalBench ALFWorld Dense Signal Dataset

ALFWorld household-task evaluation points, visual state observations, and Monte Carlo dense-signal labels under a scripted policy.

This repository contains one SignalBench dataset with two synchronized views:

• —runtime/dataset.pkl is the executable artifact used by the SignalBench benchmark code.
• —data/examples.parquet has exactly one row per benchmark example, with state/action/next-state text, renderable state_image and next_state_image columns when images exist, pointwise labels, and nested ranking candidates in ranking_candidates_json.
• —images/, when present, contains the image files referenced by the example rows.

Ground-truth labels are Monte Carlo estimates from the GT prediction JSON files under gt/. This release includes only max-aggregated Q-value and state-value labels, including pointwise and ranking variants. The Parquet export is for inspection and metadata; use the pickle for running the benchmark with SignalBench.

Counts

{
  "evaluation_points": 100,
  "examples": 100,
  "ranking_candidates": 377,
  "ranking_points": 100,
  "scalar_label_values": 954,
  "state_image_references": 200,
  "trajectory_returns": 39,
  "unique_images": 110
}

Provenance

Derived from:

• —https://github.com/alfworld/alfworld
• —https://github.com/allenai/ai2thor

Upstream license notes:

• —ALFWorld is MIT.
• —AI2-THOR is Apache-2.0.