neurips2026-anonymous/signalbench-terminalbench-bkp

SignalBench TerminalBench Dense Signal Dataset

TerminalBench evaluation points and Monte Carlo Q-value labels for studying dense signal functions in text-only terminal tasks.

This repository contains one SignalBench dataset with two synchronized views:

• —runtime/dataset.pkl is the executable artifact used by the SignalBench benchmark code.
• —data/examples.parquet has exactly one row per benchmark example, with state/action/next-state text, renderable state_image and next_state_image columns when images exist, pointwise labels, and nested ranking candidates in ranking_candidates_json.
• —images/, when present, contains the image files referenced by the example rows.

Ground-truth labels are Monte Carlo estimates from the GT prediction JSON files under gt/. This release includes only max-aggregated Q-value and state-value labels, including pointwise and ranking variants. The Parquet export is for inspection and metadata; use the pickle for running the benchmark with SignalBench.

Counts

{
  "evaluation_points": 100,
  "examples": 100,
  "ranking_candidates": 400,
  "ranking_points": 100,
  "scalar_label_values": 500,
  "state_image_references": 0,
  "trajectory_returns": 100,
  "unique_images": 0
}

Provenance

Derived from:

• —https://github.com/laude-institute/terminal-bench
• —https://github.com/harbor-framework/harbor

Upstream license notes:

• —Terminal-Bench is Apache-2.0.