forseasons/dataagentbench-derived-official54-altimate-prompt-shell

DataAgentBench-Derived Official54 Altimate Prompt Shell

This public dataset contains 54 DataAgentBench-derived task prompt-shell rows compiled for the Altimate-centered DAB sandbox runtime. It is a derived compile/export artifact, not the official raw DataAgentBench release.

It is intended as a portable task/prompt/manifest source for downstream Altimate teacher rollouts, SFT construction, or RL data compilation. It is not a completed rollout dataset: the rows do not contain agent trajectories, materialized answers, or realized rewards yet.

Files

• —rl/altimate_noctx/prompt_shell.parquet: canonical VERL-style prompt shell rows.
• —rl/altimate_noctx/prompt_shell.jsonl: JSONL copy of the same rows.
• —metadata/summary.json: discovery and selection summary.
• —metadata/readiness_report.json: readiness contract and caveats.
• —metadata/selection_report.json: selected task/domain/operation distribution.
• —metadata/discovery_report.json: discovered task-signature information.
• —metadata/selected_task_list.json: selected task identities and context.
• —metadata/sandbox_task_manifest.json: task manifest shared by RL/SFT rollouts.

Schema

Each row has the following columns:

• —level
• —type
• —data_source
• —prompt
• —ability
• —reward_model
• —extra_info

The reward_model includes the strict validator path. The extra_info field stores task identity, database context, valid DB names, operation tags, and the Altimate runtime contract.

Runtime Contract

All rows use:

• —data_source: dab_sandbox_altimate_noctx
• —dab_agent_loop_variant: altimate_noctx
• —auto_context_mode: none
• —max_iterations: 200
• —max_function_calls_per_turn: 10
• —max_single_observation_chars: 24000
• —max_turn_observation_chars: 48000
• —large_preview_chars: 16000
• —large_preview_rows: 12

Paths were sanitized from the local machine into portable DataAgentBench/... style paths. Consumers should point DATAAGENTBENCH_ROOT or equivalent runtime configuration at their local checkout of DataAgentBench task artifacts.

Quality Snapshot

• —Rows: 54
• —Task/source kind: official_like / DataAgentBench-derived
• —Readiness: ready_for_live_altimate_teacher_rollout
• —Answers materialized: False
• —Trajectories materialized: False
• —RL rows are prompt shells, not realized reward rollouts: True

Domain distribution:

• —stockmarket: 5
• —civic_unstructured: 4
• —cve: 4
• —imdb: 4
• —krama: 4
• —usaspending: 4
• —crmarenapro: 3
• —stockindex: 3
• —music_brainz_20k: 3
• —bookreview: 3
• —PATENTS: 3
• —googlelocal: 2
• —yelp: 2
• —agnews: 2
• —GITHUB_REPOS: 2
• —DEPS_DEV_V1: 2
• —PANCANCER_ATLAS: 2
• —eval46_crmarenapro: 2

Operation tags are available in extra_info.operation_tags and summarized in metadata/selection_report.json / metadata/summary.json.

Relation to DataAgentBench

This dataset is built from selected DataAgentBench task artifacts such as user queries, database configs, db_description_withhint.txt, and validate.py paths, then converted into the local Altimate/VERL prompt-shell schema. It does not redistribute the full benchmark database environment as a standalone replacement for the upstream DataAgentBench repository. Please cite the DAB paper when using this artifact.

@article{ma2026can,
  title={Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents},
  author={Ma, Ruiying and Shankar, Shreya and Chen, Ruiqi and Lin, Yiming and Zeighami, Sepanta and Ghosh, Rajoshi and Gupta, Abhinav and Gupta, Anushrut and Gopal, Tanmai and Parameswaran, Aditya G},
  journal={arXiv preprint arXiv:2603.20576},
  year={2026}
}

Intended Use

Use this dataset to launch Altimate-native teacher rollouts or compile downstream SFT/RL artifacts that share the same sandbox task manifest and validator/reward path.

A typical pipeline is:

prompt shell + sandbox_task_manifest
  -> Altimate no-context sandbox rollout
  -> strict validator / reward check
  -> SFT trajectory selection or RL training rows

Caveats

Because this is a prompt-shell dataset, reward_model.ground_truth is intentionally empty and teacher_rollout_required=true for all rows. Do not treat it as finished RL trajectory data without running the sandbox harness and validators.