Babelscape/PDDL2PRM

PDDL2PRM: Planning-Based Step-Level Supervision for Process Reward Models

PDDL2PRM is a large-scale dataset for training and evaluating Process Reward Models (PRMs) with fine-grained, step-level supervision derived from symbolic planning problems.

Unlike many PRM datasets that rely on human annotation, LLM judges, or final-answer correctness, PDDL2PRM uses Planning Domain Definition Language (PDDL) problems to generate structured reasoning trajectories whose intermediate steps can be evaluated automatically. Each reasoning step is assigned a scalar reward according to rule-based criteria that reflect whether the corresponding action is executable, goal-preserving, efficient, and optimal.

The dataset is introduced in:

Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards Raffaele Pisano and Roberto Navigli, ACL 2026

🔗 Project page & paper: https://babelscape.github.io/prm-meets-planning/ 📄 arXiv: https://arxiv.org/abs/2604.17957

Why PDDL2PRM?

Process Reward Models aim to evaluate reasoning step by step, rather than judging only the final answer. This is crucial because language models can produce a correct final answer while still making invalid, inconsistent, or unsupported intermediate reasoning steps (Turpin et al., 2023; Lightman et al., 2024; Zheng et al., 2025; Molfese et al., 2026).

However, high-quality step-level supervision is difficult to obtain:

• —manual annotation is expensive and hard to scale;
• —LLM-based annotation can be noisy and computationally costly;
• —many existing PRM datasets focus primarily on mathematics;
• —binary or coarse labels provide limited information about how a reasoning step fails.

PDDL2PRM addresses these limitations by using planning problems as a source of precise, scalable, and reproducible supervision. In PDDL, states, actions, preconditions, effects, and goals are explicitly defined, making it possible to assign rewards to reasoning steps using deterministic rules rather than subjective annotation.

Dataset Overview

PDDL2PRM contains:

Each example consists of a planning problem expressed in natural language, a partial reasoning trajectory, and a reward assigned to an intermediate step.

The dataset covers a diverse set of planning domains, including object manipulation, transportation, navigation, puzzles, and constraint-satisfaction tasks.

Domains

PDDL2PRM includes 11 planning domains:

The domains are grouped into broad reasoning families:

• —Manipulation and rearrangement: BlocksWorld variants.
• —Transportation: Ferry, Logistics, Elevator.
• —Puzzles and constraints: Tower of Hanoi, N-Puzzle.
• —Navigation and exploration: VisitGrid, Sokoban, Rooms, Spanner.

High-quality subset

We also release a smaller, high-quality subset (train_highquality.json) intended for lighter-weight training and ablations. For each problem in the training split it keeps the optimal solution trajectory plus up to 5 alternative candidate steps, sampled at evenly spaced points along the plan and prioritising the most informative rewards (partial rewards first, then non-executable, then optimal).

It shares the same schema as the full dataset.

Reward Signal

Each reasoning step is assigned one of five scalar rewards:

This reward structure provides more information than binary correct/incorrect supervision. It distinguishes between different types of flawed reasoning, including invalid actions, irreversible mistakes, inefficient detours, and merely suboptimal decisions.

How the Dataset Was Generated

PDDL2PRM is generated from symbolic planning problems through the following pipeline:

1. 1.Parse PDDL domains and problems. Planning instances define states, actions, preconditions, effects, initial states, and goals.

1. 1.Sample candidate actions at each state. For each state along a trajectory, multiple candidate actions are considered, including valid, invalid, suboptimal, and optimal actions.

1. 1.Evaluate each action with planning-based rules. Actions are labeled according to executability, reachability of the goal, backtracking behavior, and membership in an optimal plan.

1. 1.Translate actions into natural language reasoning steps. Each symbolic action is converted into a natural-language reasoning step and paired with its scalar reward.

1. 1.Continue until the goal is reached. The procedure produces both correct and incorrect reasoning trajectories, enabling PRMs to learn fine-grained distinctions between different reasoning behaviors.

Key Findings from the Paper

Training with PDDL2PRM improves PRM robustness and generalization.

In the paper, PRMs trained on datasets augmented with PDDL-derived supervision outperform comparable models trained without the PDDL component. Improvements are especially strong on non-mathematical and planning-based reasoning benchmarks.

The results suggest that planning-derived supervision helps PRMs detect not only local execution errors, but also deeper modeling errors such as invalid assumptions, inappropriate reductions, and unjustified constraints.

Dataset Card Note

PDDL2PRM is released under the CC BY-SA 4.0 license.

Because the dataset is derived from symbolic planning problems and template-based natural-language translations, users should consider it a high-precision source of structured reasoning supervision rather than a replacement for naturally occurring Chain-of-Thought data.

Citation

If you use PDDL2PRM, please cite:

@inproceedings{prmsmeetplanning2026,
  title={Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards},
  author={Pisano, Raffaele and Navigli, Roberto},
  booktitle={Proceedings of ACL 2026},
  year={2026}
}