Babelscape/PDDL2PRM
收藏资源简介：
--- license: cc-by-sa-4.0 language: - en task_categories: - text-classification - reinforcement-learning tags: - process-reward-models - prm - reasoning - chain-of-thought - planning - pddl - step-level-supervision - synthetic-data - logical-reasoning - reward-modeling pretty_name: PDDL2PRM size_categories: - 100K<n<1M --- # PDDL2PRM: Planning-Based Step-Level Supervision for Process Reward Models **PDDL2PRM** is a large-scale dataset for training and evaluating **Process Reward Models (PRMs)** with fine-grained, step-level supervision derived from symbolic planning problems. Unlike many PRM datasets that rely on human annotation, LLM judges, or final-answer correctness, PDDL2PRM uses **Planning Domain Definition Language (PDDL)** problems to generate structured reasoning trajectories whose intermediate steps can be evaluated automatically. Each reasoning step is assigned a scalar reward according to rule-based criteria that reflect whether the corresponding action is executable, goal-preserving, efficient, and optimal. The dataset is introduced in: **Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards** Raffaele Pisano and Roberto Navigli, ACL 2026 🔗 **Project page & paper:** https://babelscape.github.io/prm-meets-planning/ 📄 **arXiv:** https://arxiv.org/abs/2604.17957 --- ## Why PDDL2PRM? Process Reward Models aim to evaluate reasoning **step by step**, rather than judging only the final answer. This is crucial because language models can produce a correct final answer while still making invalid, inconsistent, or unsupported intermediate reasoning steps (Turpin et al., 2023; Lightman et al., 2024; Zheng et al., 2025; Molfese et al., 2026). However, high-quality step-level supervision is difficult to obtain: - manual annotation is expensive and hard to scale; - LLM-based annotation can be noisy and computationally costly; - many existing PRM datasets focus primarily on mathematics; - binary or coarse labels provide limited information about *how* a reasoning step fails. PDDL2PRM addresses these limitations by using planning problems as a source of **precise, scalable, and reproducible supervision**. In PDDL, states, actions, preconditions, effects, and goals are explicitly defined, making it possible to assign rewards to reasoning steps using deterministic rules rather than subjective annotation. --- ## Dataset Overview PDDL2PRM contains: | Statistic | Value | |---|---:| | Reasoning steps | **984,974** | | Planning problems | **14,714** | | Domains | **11** | | Reward levels | **5** | | Mean optimal plan length | **7.94** | Each example consists of a planning problem expressed in natural language, a partial reasoning trajectory, and a reward assigned to an intermediate step. The dataset covers a diverse set of planning domains, including object manipulation, transportation, navigation, puzzles, and constraint-satisfaction tasks. --- ## Domains PDDL2PRM includes 11 planning domains: | Domain | Problems | Total Steps | |---|---:|---:| | BlocksWorld-3 | 952 | 51,990 | | BlocksWorld-4 | 1,796 | 125,402 | | Ferry | 858 | 65,269 | | Hanoi | 1,660 | 65,745 | | Logistics | 669 | 70,238 | | Elevator | 2,089 | 207,549 | | N-Puzzle | 598 | 42,027 | | Rooms | 916 | 37,947 | | Sokoban | 437 | 58,475 | | Spanner | 3,563 | 198,942 | | VisitGrid | 1,176 | 61,390 | | **Total** | **14,714** | **984,974** | The domains are grouped into broad reasoning families: - **Manipulation and rearrangement:** BlocksWorld variants. - **Transportation:** Ferry, Logistics, Elevator. - **Puzzles and constraints:** Tower of Hanoi, N-Puzzle. - **Navigation and exploration:** VisitGrid, Sokoban, Rooms, Spanner. --- ## Reward Signal Each reasoning step is assigned one of five scalar rewards: | Reward | Label | Meaning | |---:|---|---| | **0.0** | Non-executable | The action cannot be applied because its preconditions are not satisfied. | | **0.25** | Dead-end | The action leads to a state from which the goal is unreachable. | | **0.5** | Backtracking | The action leads to a state from which an optimal plan requires revisiting a previous state. | | **0.75** | Suboptimal | The action is valid but does not belong to any optimal plan. | | **1.0** | Optimal | The action belongs to at least one optimal plan from the current state. | This reward structure provides more information than binary correct/incorrect supervision. It distinguishes between different types of flawed reasoning, including invalid actions, irreversible mistakes, inefficient detours, and merely suboptimal decisions. --- ## How the Dataset Was Generated PDDL2PRM is generated from symbolic planning problems through the following pipeline: 1. **Parse PDDL domains and problems.** Planning instances define states, actions, preconditions, effects, initial states, and goals. 2. **Sample candidate actions at each state.** For each state along a trajectory, multiple candidate actions are considered, including valid, invalid, suboptimal, and optimal actions. 3. **Evaluate each action with planning-based rules.** Actions are labeled according to executability, reachability of the goal, backtracking behavior, and membership in an optimal plan. 4. **Translate actions into natural language reasoning steps.** Each symbolic action is converted into a natural-language reasoning step and paired with its scalar reward. 5. **Continue until the goal is reached.** The procedure produces both correct and incorrect reasoning trajectories, enabling PRMs to learn fine-grained distinctions between different reasoning behaviors. --- ## Key Findings from the Paper Training with PDDL2PRM improves PRM robustness and generalization. In the paper, PRMs trained on datasets augmented with PDDL-derived supervision outperform comparable models trained without the PDDL component. Improvements are especially strong on non-mathematical and planning-based reasoning benchmarks. The results suggest that planning-derived supervision helps PRMs detect not only local execution errors, but also deeper modeling errors such as invalid assumptions, inappropriate reductions, and unjustified constraints. --- ## Dataset Card Note PDDL2PRM is released under the **CC BY-SA 4.0** license. Because the dataset is derived from symbolic planning problems and template-based natural-language translations, users should consider it a high-precision source of structured reasoning supervision rather than a replacement for naturally occurring Chain-of-Thought data. --- ## Citation If you use PDDL2PRM, please cite: ```bibtex @inproceedings{prmsmeetplanning2026, title={Process Reward Models Meet Planning: Generating Precise and Scalable Datasets for Step-Level Rewards}, author={Pisano, Raffaele and Navigli, Roberto}, booktitle={Proceedings of ACL 2026}, year={2026} }
PDDL2PRM is a large-scale dataset for training and evaluating Process Reward Models (PRMs) with fine-grained, step-level supervision derived from symbolic planning problems using the Planning Domain Definition Language (PDDL). Unlike many PRM datasets that rely on human annotation, LLM judges, or final-answer correctness, PDDL2PRM uses PDDL problems to generate structured reasoning trajectories whose intermediate steps can be evaluated automatically. Each reasoning step is assigned a scalar reward according to rule-based criteria that reflect whether the corresponding action is executable, goal-preserving, efficient, and optimal. The dataset includes 984,974 reasoning steps, 14,714 planning problems, and covers 11 domains (e.g., object manipulation, transportation, navigation, puzzles), with 5 reward levels (from 0.0 to 1.0) indicating different quality of actions. It provides precise, scalable, and reproducible supervision through planning problems, aiming to help PRMs distinguish various reasoning errors and improve robustness and generalization on non-mathematical and planning-based reasoning tasks.
