PMLRM-Bench: An Object-Centric Event Log of Large Reasoning Model Reasoning Steps for Process Mining Analysis

This dataset provides an object-centric event log (OCEL) detailing the reasoning processes of various Large Reasoning Models (LRMs) when tackling tasks from the PMLRM-Bench benchmark. The PMLRM-Bench is an extension of the PM-LLM-Benchmark, designed to evaluate both the correctness of LRM outputs and the robustness of their reasoning processes in the domain of process mining. Link to the benchmark's repository: https://github.com/fit-alessandro-berti/pmllmbench-lrms-reasoning-analysis/ Link to the benchmark's (pre-print) paper: https://www.alessandroberti.it/new_papers/2025_Berti_Configuring_LRMs.pdf The OCEL is generated from the textual "chain-of-thought" outputs of LRMs. Each reasoning step within these traces has been extracted and classified by its type (e.g., Deductive Reasoning, Hypothesis Generation) and its effect on the overall reasoning correctness (Positive, Indifferent, or Negative). This classification was performed using a judge LLM (Gemini-2.5-Pro-Preview-03-25), as detailed in the source paper. Structure of the OCEL: The event log is structured with the following object types and event attributes: Objects: MOD: Represents a specific Large Reasoning Model evaluated in the benchmark. QUE: Represents a unique question or prompt from the PM-LLM-Benchmark dataset that the LRM responded to. MODQUE: Represents a unique instance of a specific model (MOD) answering a specific question (QUE). Events: Each event corresponds to a single reasoning step identified in the LRM's output. ocel:activity: Stores the classified reasoning step, combining its type (e.g., PR, DR, HG) and its effect (PE, IND, NE). For example, "Deductive Reasoning - PE". ocel:timestamp: A synthetically generated timestamp to preserve the order of reasoning steps within a trace. text: Contains the actual text snippet from the LRM's reasoning trace that corresponds to this specific step. ocel:eid: A unique identifier for the event. Relations: Each event is linked to: The MOD object that produced the reasoning step. The QUE object that the reasoning step is addressing. The MODQUE object representing the specific answer instance. Purpose and Potential Use: This OCEL allows for in-depth analysis of LRM reasoning behaviors using process mining techniques. Researchers can explore: Common reasoning patterns across different models or question types. The sequence and frequency of various reasoning steps (e.g., how often Hypothesis Generation is followed by Validation). The impact of different reasoning strategies on task performance and correctness. Differences in reasoning approaches between high-performing and lower-performing LRMs. The dataset is intended to complement the research paper "Configuring Large Reasoning Models using Process Mining: A Benchmark and a Case Study" by Berti et al., providing the structured data used to analyze and benchmark LRM reasoning capabilities. File Information: The dataset contains one file: reasoning_benchmark.jsonocel. This file is an object-centric event log formatted according to the OCEL 2.0 standard. This dataset was generated using a Python script that parses the JSON files containing the classified reasoning steps (from the prel/final_abstract_steps folder mentioned in the script, which corresponds to the outputs of the reasoning trace extraction and classification pipeline described in Section 3.1 of the paper ).本数据集提供一份面向对象事件日志（Object-Centric Event Log，OCEL），详细记录各类大推理模型（Large Reasoning Models，LRMs）在求解PMLRM-Bench基准测试任务时的推理过程。PMLRM-Bench是PM-LLM-Benchmark的扩展版本，专为在过程挖掘（Process Mining）领域评测大推理模型输出的正确性与推理过程的鲁棒性而设计。 基准测试仓库链接：https://github.com/fit-alessandro-berti/pmllmbench-lrms-reasoning-analysis/ 基准测试预印本论文链接：https://www.alessandroberti.it/new_papers/2025_Berti_Configuring_LRMs.pdf 该OCEL源自大推理模型的文本式思维链（Chain-of-Thought）输出。上述轨迹中的每一步推理均已被提取，并按其类型（如演绎推理（Deductive Reasoning）、假设生成（Hypothesis Generation））以及对整体推理正确性的影响（正面、中性或负面）完成分类。该分类工作由评判型大语言模型（Gemini-2.5-Pro-Preview-03-25）完成，详情参见源论文。 OCEL的结构： 对象类型： MOD：代表基准中评估的特定大推理模型。 QUE：代表PM-LLM-Benchmark数据集中LRM需响应的唯一问题或提示词。 MODQUE：代表特定模型（MOD）回答特定问题（QUE）的唯一实例。 事件： 每个事件对应LRM输出中识别出的单步推理。 - ocel:activity：存储分类后的推理步骤，整合其类型（如PR、DR、HG）与影响标识（PE、IND、NE），例如"Deductive Reasoning - PE"。 - ocel:timestamp：人工生成的时间戳，用于保留轨迹内推理步骤的先后顺序。 - text：包含与该特定推理步骤对应的LRM推理轨迹中的原始文本片段。 - ocel:eid：该事件的唯一标识符。 关联关系： 每个事件均关联以下对象： 1. 生成该推理步骤的MOD对象 2. 该推理步骤所针对的QUE对象 3. 代表该具体回答实例的MODQUE对象 用途与潜在应用场景： 本OCEL支持借助过程挖掘技术对LRM的推理行为开展深度分析。研究人员可探索以下方向： - 不同模型或问题类型下的通用推理模式 - 各类推理步骤的序列与出现频次（例如假设生成步骤后紧随验证步骤的频率） - 不同推理策略对任务性能与正确性的影响 - 高性能与低性能LRM之间推理方法的差异 本数据集旨在补充Berti等人的研究论文"Configuring Large Reasoning Models using Process Mining: A Benchmark and a Case Study"，提供用于分析与评测LRM推理能力的结构化数据。 文件信息： 本数据集仅包含一个文件：reasoning_benchmark.jsonocel。该文件为符合OCEL 2.0标准的面向对象事件日志。 本数据集通过Python脚本生成，该脚本可解析包含分类后推理步骤的JSON文件（来自脚本提及的prel/final_abstract_steps文件夹，对应论文第3.1节所述的推理轨迹提取与分类流水线的输出）。