PMLRM-Bench: An Object-Centric Event Log of Large Reasoning Model Reasoning Steps for Process Mining Analysis

This dataset provides an object-centric event log (OCEL) detailing the reasoning processes of various Large Reasoning Models (LRMs) when tackling tasks from the PMLRM-Bench benchmark. The PMLRM-Bench is an extension of the PM-LLM-Benchmark, designed to evaluate both the correctness of LRM outputs and the robustness of their reasoning processes in the domain of process mining. Link to the benchmark's repository: https://github.com/fit-alessandro-berti/pmllmbench-lrms-reasoning-analysis/ Link to the benchmark's (pre-print) paper: https://www.alessandroberti.it/new_papers/2025_Berti_Configuring_LRMs.pdf The OCEL is generated from the textual "chain-of-thought" outputs of LRMs. Each reasoning step within these traces has been extracted and classified by its type (e.g., Deductive Reasoning, Hypothesis Generation) and its effect on the overall reasoning correctness (Positive, Indifferent, or Negative). This classification was performed using a judge LLM (Gemini-2.5-Pro-Preview-03-25), as detailed in the source paper. Structure of the OCEL: The event log is structured with the following object types and event attributes: Objects: MOD: Represents a specific Large Reasoning Model evaluated in the benchmark. QUE: Represents a unique question or prompt from the PM-LLM-Benchmark dataset that the LRM responded to. MODQUE: Represents a unique instance of a specific model (MOD) answering a specific question (QUE). Events: Each event corresponds to a single reasoning step identified in the LRM's output. ocel:activity: Stores the classified reasoning step, combining its type (e.g., PR, DR, HG) and its effect (PE, IND, NE). For example, "Deductive Reasoning - PE". ocel:timestamp: A synthetically generated timestamp to preserve the order of reasoning steps within a trace. text: Contains the actual text snippet from the LRM's reasoning trace that corresponds to this specific step. ocel:eid: A unique identifier for the event. Relations: Each event is linked to: The MOD object that produced the reasoning step. The QUE object that the reasoning step is addressing. The MODQUE object representing the specific answer instance. Purpose and Potential Use: This OCEL allows for in-depth analysis of LRM reasoning behaviors using process mining techniques. Researchers can explore: Common reasoning patterns across different models or question types. The sequence and frequency of various reasoning steps (e.g., how often Hypothesis Generation is followed by Validation). The impact of different reasoning strategies on task performance and correctness. Differences in reasoning approaches between high-performing and lower-performing LRMs. The dataset is intended to complement the research paper "Configuring Large Reasoning Models using Process Mining: A Benchmark and a Case Study" by Berti et al., providing the structured data used to analyze and benchmark LRM reasoning capabilities. File Information: The dataset contains one file: reasoning_benchmark.jsonocel. This file is an object-centric event log formatted according to the OCEL 2.0 standard. This dataset was generated using a Python script that parses the JSON files containing the classified reasoning steps (from the prel/final_abstract_steps folder mentioned in the script, which corresponds to the outputs of the reasoning trace extraction and classification pipeline described in Section 3.1 of the paper ).本数据集提供了一个面向对象的事件日志（Object-Centric Event Log，OCEL），详细记录了各类大推理模型（Large Reasoning Models, LRMs）在处理PMLRM-Bench基准测试任务时的推理过程。PMLRM-Bench是PM-LLM-Benchmark的扩展版本，旨在评估流程挖掘领域内大推理模型输出的正确性，以及其推理过程的鲁棒性。 基准测试仓库链接：https://github.com/fit-alessandro-berti/pmllmbench-lrms-reasoning-analysis/ 基准测试预印本论文链接：https://www.alessandroberti.it/new_papers/2025_Berti_Configuring_LRMs.pdf 该OCEL源自大推理模型的文本型思维链（Chain-of-Thought）输出。研究人员已提取这些推理轨迹中的每一步推理，并按其类型（如演绎推理、假设生成）以及对整体推理正确性的影响（正向、中性或负向）进行分类。该分类工作由裁判大语言模型（Gemini-2.5-Pro-Preview-03-25）完成，详情参见源论文。 OCEL结构： 本事件日志包含以下对象类型与事件属性： ### 对象（Objects） MOD：代表基准测试中被评估的某一特定大推理模型。 QUE：代表PM-LLM-Benchmark数据集中大推理模型所响应的唯一问题或提示词。 MODQUE：代表某一特定模型（MOD）应答某一特定问题（QUE）的唯一实例。 ### 事件（Events） 每个事件对应大推理模型输出中识别出的单步推理。 - ocel:activity：存储已分类的推理步骤，整合其类型（如PR、DR、HG）及其影响（PE、IND、NE），例如“演绎推理 - PE”。 - ocel:timestamp：为合成生成的时间戳，用于保留推理轨迹内各步骤的顺序。 - text：包含与该特定步骤对应的大推理模型推理轨迹中的实际文本片段。 - ocel:eid：事件的唯一标识符。 ### 关联关系（Relations） 每个事件均关联至以下对象： 1. 生成该推理步骤的MOD对象。 2. 该推理步骤所针对的QUE对象。 3. 代表该特定应答实例的MODQUE对象。 ### 用途与潜在应用场景 本OCEL可借助流程挖掘技术，对大推理模型的推理行为开展深度分析。研究人员可探索以下内容： - 不同模型或不同问题类型下的通用推理模式。 - 各类推理步骤的序列与出现频率（例如，假设生成步骤后紧跟验证步骤的频次）。 - 不同推理策略对任务性能与正确性的影响。 - 高性能与低性能大推理模型之间推理方法的差异。 本数据集旨在作为Berti等人的研究论文《基于流程挖掘配置大推理模型：基准测试与案例研究》的补充资料，提供用于分析与评估大推理模型推理能力的结构化数据。 ### 文件信息 本数据集仅包含一个文件：reasoning_benchmark.jsonocel。该文件为遵循OCEL 2.0标准格式的面向对象事件日志。 本数据集通过Python脚本生成，该脚本会解析包含已分类推理步骤的JSON文件（脚本中提及的prel/final_abstract_steps文件夹，对应论文3.1节所述的推理轨迹提取与分类流水线的输出结果）。