JetBrains-Research/REval

--- license: mit task_categories: - text-generation - question-answering language: - en - code tags: - code - program-analysis - runtime-behavior - execution-traces - code-reasoning - python pretty_name: "REval: Reasoning Evaluation" size_categories: - n<1K dataset_info: - config_name: problems features: - name: task_id dtype: string - name: code dtype: string - name: entry_point dtype: string - name: test dtype: string - name: inputs sequence: string - name: outputs sequence: string splits: - name: test num_examples: 154 - config_name: tasks features: - name: task_id dtype: string - name: idx dtype: int32 - name: tasks dtype: string splits: - name: test num_examples: 154 - config_name: executions features: - name: task_id dtype: string - name: idx dtype: int32 - name: input_idx dtype: int32 - name: problem_type dtype: string - name: input dtype: string - name: expected_output dtype: string - name: actual_output dtype: string - name: status dtype: string - name: trace sequence: int32 - name: coverage sequence: int32 - name: num_states dtype: int32 - name: code_hash dtype: string - name: error dtype: string splits: - name: test num_examples: 694 - config_name: states features: - name: task_id dtype: string - name: idx dtype: int32 - name: input_idx dtype: int32 - name: states dtype: string splits: - name: test num_examples: 694 configs: - config_name: problems data_files: - split: test path: "data/problems.jsonl" default: true - config_name: tasks data_files: - split: test path: "data/tasks.jsonl" - config_name: executions data_files: - split: test path: "data/executions.jsonl" - config_name: states data_files: - split: test path: "data/states.jsonl" --- # REval: Reasoning Runtime Behavior of a Program with LLM > **Disclaimer:** We are not the authors of the REval benchmark. This upload is a convenience repackaging of the original dataset with precomputed execution traces, variable states, and ground truth answers to make the benchmark easier to use programmatically. The original benchmark was created by Junkai Chen et al. and is available at [github.com/r-eval/REval](https://github.com/r-eval/REval/). Please cite the original paper if you use this data. REval is a benchmark for evaluating Large Language Models' ability to reason about the **runtime behavior** of Python programs. > **Reasoning Runtime Behavior of a Program with LLM: How Far Are We?** > Junkai Chen, Zhiyuan Pan, Xing Hu, Zhenhao Li, Ge Li, Xin Xia > ICSE 2025 > [Paper](https://doi.org/10.1109/ICSE55347.2025.00087) | [GitHub](https://github.com/r-eval/REval/) ## Dataset Summary | | | |---|---| | **Problems** | 154 (85 HumanEval + 69 ClassEval) | | **Test-case executions** | 694 (with full execution traces) | | **Reasoning tasks** | Coverage, Path, State, Output, Consistency | | **Ground truth** | Line coverage, execution traces, variable states at every step | | **License** | MIT | ## Reasoning Tasks 1. **Coverage** -- Predict whether a specific line of code is executed for a given input 2. **Path** -- Determine the next line that will be executed after a given line 3. **State** -- Infer variable values at specific execution points 4. **Output** -- Complete test code based on expected execution behavior 5. **Consistency** -- Combined score measuring consistency across all four tasks ## Configurations | Config | Records | Description | |--------|---------|-------------| | `problems` (default) | 154 | Problem definitions: code, inputs, expected outputs | | `tasks` | 154 | Task specifications: which lines/variables to query per input | | `executions` | 694 | Execution traces and line coverage per (problem, input) pair | | `states` | 694 | Variable states at each executed line | ## Usage ```python from datasets import load_dataset # Load problem definitions (default config) problems = load_dataset("r-eval/REval", "problems", split="test") print(f"{len(problems)} problems") print(problems[0]["task_id"]) # "DREval/0" print(problems[0]["entry_point"]) # "has_close_elements" # Load execution traces executions = load_dataset("r-eval/REval", "executions", split="test") print(executions[0]["trace"]) # [11, 12, 13, 12, 13, 14, 15, ...] print(executions[0]["coverage"]) # [11, 12, 13, 14, 15, 16] # Load variable states (states field is a JSON string -- parse it) import json states = load_dataset("r-eval/REval", "states", split="test") state_list = json.loads(states[0]["states"]) # Each state: {"lineno": 0, "locals": {"var": {"__type__": "int", "__value__": 1}}} # Load task specifications (tasks field is a JSON string) tasks = load_dataset("r-eval/REval", "tasks", split="test") task_list = json.loads(tasks[0]["tasks"]) # Each task: {"input_idx": 0, "task": [{"lineno": 17, "var": "distance"}, ...], "output_pred": "..."} ``` ## Data Fields ### `problems` config | Field | Type | Description | |-------|------|-------------| | `task_id` | string | Unique identifier, e.g. `"DREval/0"` | | `code` | string | Complete Python source code (signature + docstring + solution) | | `entry_point` | string | Function or class name | | `test` | string or null | Unittest code for ClassEval problems; null for HumanEval | | `inputs` | list[string] | Test inputs as Python expressions | | `outputs` | list[string] | Expected outputs as strings | **Problem types:** - **HumanEval** (idx 0--84): Standalone functions. `test` is null, `outputs` is non-empty. - **ClassEval** (idx 85--153): OOP classes. `test` contains unittest code, `outputs` is empty. ### `tasks` config | Field | Type | Description | |-------|------|-------------| | `task_id` | string | Unique identifier | | `idx` | int | Problem index | | `tasks` | string (JSON) | JSON-encoded list of per-input task definitions | Each entry in the parsed `tasks` list contains: - `input_idx` (int): Index into the problem's inputs/outputs arrays - `task` (list[object]): Variable queries -- each has `lineno` (1-indexed) and `var` (variable name) - `output_pred` (string): Output prediction template (e.g. `"assert func(args) == ??"`) ### `executions` config | Field | Type | Description | |-------|------|-------------| | `task_id` | string | Problem identifier | | `idx` | int | Problem index | | `input_idx` | int | Which test input was used | | `problem_type` | string | `"humaneval"` or `"classeval"` | | `input` | string | The specific input expression | | `expected_output` | string | Expected output | | `actual_output` | string | Actual output from execution | | `status` | string | `"ok"` or `"error"` | | `trace` | list[int] | 0-indexed line execution sequence | | `coverage` | list[int] | Sorted unique executed lines (0-indexed) | | `num_states` | int | Number of state snapshots captured | | `code_hash` | string | SHA-256 of the source code | | `error` | string or null | Error message if `status="error"`, null otherwise | ### `states` config | Field | Type | Description | |-------|------|-------------| | `task_id` | string | Problem identifier | | `idx` | int | Problem index | | `input_idx` | int | Which test input was used | | `states` | string (JSON) | JSON-encoded list of state objects | Each state object in the parsed list: - `lineno` (int): 0-indexed line number - `locals` (dict): Variable name to typed value envelope (`{"__type__": "int", "__value__": 42}`) - `return` (optional): Return value in the same envelope format - `exception` (optional): Exception info if one was raised **Supported value types in envelopes:** `int`, `float`, `bool`, `str`, `NoneType`, `list`, `tuple`, `set`, `dict`, `Nil` (uninitialized), `numpy.ndarray`, `datetime.datetime`, and custom objects. Special float values: `"nan"`, `"inf"`, `"-inf"`. ## Line Number Conventions - **`executions` config** (`trace`, `coverage`): **0-indexed** line numbers - **`states` config** (`lineno`): **0-indexed** line numbers - **`tasks` config** (`lineno` in task queries): **1-indexed** line numbers (for use in prompts) ## Known Issues - Problems `DREval/117` and `DREval/149` import `gensim` (not included in dependencies). Their ground truth records have `status="error"` with empty traces. - Arrows to the next executed lines for ClassEval do not take into account test code. ## Citation If you use this dataset, please cite: ```bibtex @inproceedings{chen2025reval, title = {Reasoning Runtime Behavior of a Program with LLM: How Far Are We?}, author = {Junkai Chen and Zhiyuan Pan and Xing Hu and Zhenhao Li and Ge Li and Xin Xia}, booktitle = {Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE)}, year = {2025}, doi = {10.1109/ICSE55347.2025.00087} } ``` ## Source - **Repository:** [github.com/r-eval/REval](https://github.com/r-eval/REval/) - **Paper:** [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://doi.org/10.1109/ICSE55347.2025.00087) (ICSE 2025) - **License:** MIT许可证：MIT 任务类别： - 文本生成 - 问答 语言： - 英语 - 代码 标签： - 代码 - 程序分析 - 运行时行为 - 执行轨迹 - 代码推理 - Python 展示名称："REval：推理评估" 规模类别： - n<1K 数据集信息： - 配置名称：problems 特征： - 名称：task_id 数据类型：string - 名称：code 数据类型：string - 名称：entry_point 数据类型：string - 名称：test 数据类型：string - 名称：inputs 序列类型：string - 名称：outputs 序列类型：string 划分： - 名称：test 示例数量：154 - 配置名称：tasks 特征： - 名称：task_id 数据类型：string - 名称：idx 数据类型：int32 - 名称：tasks 数据类型：string 划分： - 名称：test 示例数量：154 - 配置名称：executions 特征： - 名称：task_id 数据类型：string - 名称：idx 数据类型：int32 - 名称：input_idx 数据类型：int32 - 名称：problem_type 数据类型：string - 名称：input 数据类型：string - 名称：expected_output 数据类型：string - 名称：actual_output 数据类型：string - 名称：status 数据类型：string - 名称：trace 序列类型：int32 - 名称：coverage 序列类型：int32 - 名称：num_states 数据类型：int32 - 名称：code_hash 数据类型：string - 名称：error 数据类型：string 划分： - 名称：test 示例数量：694 - 配置名称：states 特征： - 名称：task_id 数据类型：string - 名称：idx 数据类型：int32 - 名称：input_idx 数据类型：int32 - 名称：states 数据类型：string 划分： - 名称：test 示例数量：694 配置项： - 配置名称：problems 数据文件： - 划分：test 路径："data/problems.jsonl" 默认：true - 配置名称：tasks 数据文件： - 划分：test 路径："data/tasks.jsonl" - 配置名称：executions 数据文件： - 划分：test 路径："data/executions.jsonl" - 配置名称：states 数据文件： - 划分：test 路径："data/states.jsonl" # REval：基于大语言模型（Large Language Model）的程序运行时行为推理 > **免责声明：** 本团队并非REval基准测试的原开发者。本次上传仅为便捷重新打包原始数据集，附带预计算的执行轨迹、变量状态与标准答案，以降低该基准测试的程序化使用门槛。原始基准测试由Junkai Chen等人创建，可在[github.com/r-eval/REval](https://github.com/r-eval/REval/)获取。若使用本数据集，请引用原始论文。 REval是一款用于评估大语言模型（Large Language Model）推理Python程序运行时行为能力的基准测试集。 > **《基于大语言模型的程序运行时行为推理：我们的水平如何？》** > Junkai Chen、Zhiyuan Pan、Xing Hu、Zhenhao Li、Ge Li、Xin Xia > ICSE 2025 > [论文](https://doi.org/10.1109/ICSE55347.2025.00087) | [GitHub仓库](https://github.com/r-eval/REval/) ## 数据集概览 | | | |---|---| | **问题集** | 154个（85个HumanEval + 69个ClassEval） | | **测试用例执行记录** | 694条（包含完整执行轨迹） | | **推理任务类型** | 覆盖度、路径、状态、输出、一致性 | | **标准答案** | 行覆盖度、执行轨迹、每一步的变量状态 | | **许可证** | MIT | ## 推理任务 1. **覆盖度**：针对给定输入，预测某一特定代码行是否会被执行 2. **路径**：确定给定代码行之后的下一条将被执行的代码行 3. **状态**：推断特定执行点处的变量值 4. **输出**：基于预期执行行为补全测试代码 5. **一致性**：衡量上述四项任务整体一致性的综合得分 ## 配置项 | 配置名称 | 记录数 | 描述 | |--------|---------|-------------| | `problems`（默认） | 154 | 问题定义：包含代码、测试输入、预期输出 | | `tasks` | 154 | 任务规范：每个测试输入对应的查询行/变量信息 | | `executions` | 694 | 每个（问题，输入）对的执行轨迹与行覆盖度 | | `states` | 694 | 每条已执行代码行对应的变量状态 | ## 使用方法 python from datasets import load_dataset # 加载问题定义（默认配置） problems = load_dataset("r-eval/REval", "problems", split="test") print(f"{len(problems)} 个问题") print(problems[0]["task_id"]) # "DREval/0" print(problems[0]["entry_point"]) # "has_close_elements" # 加载执行轨迹 executions = load_dataset("r-eval/REval", "executions", split="test") print(executions[0]["trace"]) # [11, 12, 13, 12, 13, 14, 15, ...] print(executions[0]["coverage"]) # [11, 12, 13, 14, 15, 16] # 加载变量状态（states字段为JSON字符串，请解析后使用） import json states = load_dataset("r-eval/REval", "states", split="test") state_list = json.loads(states[0]["states"]) # 每个状态对象格式：{"lineno": 0, "locals": {"var": {"__type__": "int", "__value__": 1}}} # 加载任务规范（tasks字段为JSON字符串） tasks = load_dataset("r-eval/REval", "tasks", split="test") task_list = json.loads(tasks[0]["tasks"]) # 每个任务格式：{"input_idx": 0, "task": [{"lineno": 17, "var": "distance"}, ...], "output_pred": "..."} ## 数据字段 ### `problems` 配置 | 字段 | 数据类型 | 描述 | |-------|------|-------------| | `task_id` | 字符串 | 唯一标识符，例如 `"DREval/0"` | | `code` | 字符串 | 完整Python源代码（包含函数签名、文档字符串与解决方案代码） | | `entry_point` | 字符串 | 函数或类名 | | `test` | 字符串或null | ClassEval问题的单元测试代码；HumanEval问题的该字段为null | | `inputs` | 字符串列表 | 以Python表达式形式给出的测试输入 | | `outputs` | 字符串列表 | 以字符串形式给出的预期输出 | **问题类型：** - **HumanEval**（索引0--84）：独立函数。`test`字段为null，`outputs`字段非空。 - **ClassEval**（索引85--153）：面向对象类。`test`字段包含单元测试代码，`outputs`字段为空。 ### `tasks` 配置 | 字段 | 数据类型 | 描述 | |-------|------|-------------| | `task_id` | 字符串 | 唯一标识符 | | `idx` | int32 | 问题索引 | | `tasks` | 字符串（JSON格式） | JSON编码的逐输入任务定义列表 | 解析后的`tasks`列表中的每个条目包含： - `input_idx`（整数）：对应问题的输入/输出数组的索引 - `task`（对象列表）：变量查询项，每个项包含`lineno`（1索引行号）与`var`（变量名） - `output_pred`（字符串）：输出预测模板（例如 `"assert func(args) == ??"`） ### `executions` 配置 | 字段 | 数据类型 | 描述 | |-------|------|-------------| | `task_id` | 字符串 | 问题标识符 | | `idx` | int32 | 问题索引 | | `input_idx` | int32 | 所用测试输入的索引 | | `problem_type` | 字符串 | `"humaneval"` 或 `"classeval"` | | `input` | 字符串 | 具体的输入表达式 | | `expected_output` | 字符串 | 预期输出 | | `actual_output` | 字符串 | 实际执行得到的输出 | | `status` | 字符串 | `"ok"` 或 `"error"` | | `trace` | int32序列 | 0索引的代码行执行序列 | | `coverage` | int32序列 | 排序后的唯一已执行代码行（0索引） | | `num_states` | int32 | 捕获的状态快照数量 | | `code_hash` | 字符串 | 源代码的SHA-256哈希值 | | `error` | 字符串或null | 若`status="error"`则为错误信息，否则为null | ### `states` 配置 | 字段 | 数据类型 | 描述 | |-------|------|-------------| | `task_id` | 字符串 | 问题标识符 | | `idx` | int32 | 问题索引 | | `input_idx` | int32 | 所用测试输入的索引 | | `states` | 字符串（JSON格式） | JSON编码的状态对象列表 | 解析后的列表中的每个状态对象： - `lineno`（整数）：0索引的代码行号 - `locals`（字典）：变量名到类型化值封装的映射，格式为`{"__type__": "int", "__value__": 42}` - `return`（可选）：返回值，采用相同的封装格式 - `exception`（可选）：若抛出异常则包含异常信息 **支持的封装值类型：** `int`、`float`、`bool`、`str`、`NoneType`、`list`、`tuple`、`set`、`dict`、`Nil`（未初始化）、`numpy.ndarray`、`datetime.datetime`以及自定义对象。特殊浮点值：`"nan"`、`"inf"`、`"-inf"`。 ## 行号约定 - **`executions` 配置**（`trace`、`coverage`字段）：**0索引**代码行号 - **`states` 配置**（`lineno`字段）：**0索引**代码行号 - **`tasks` 配置**（任务查询中的`lineno`）：**1索引**代码行号（用于提示词中） ## 已知问题 - 问题`DREval/117`与`DREval/149`导入了`gensim`库（未包含在依赖项中）。它们的标准答案记录的`status`为`"error"`且轨迹为空。 - ClassEval的下一条执行行箭头未考虑测试代码。 ## 引用 若使用本数据集，请引用： bibtex @inproceedings{chen2025reval, title = {Reasoning Runtime Behavior of a Program with LLM: How Far Are We?}, author = {Junkai Chen and Zhiyuan Pan and Xing Hu and Zhenhao Li and Ge Li and Xin Xia}, booktitle = {Proceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE)}, year = {2025}, doi = {10.1109/ICSE55347.2025.00087} } ## 来源 - **仓库地址**：[github.com/r-eval/REval](https://github.com/r-eval/REval/) - **论文**：[《基于大语言模型的程序运行时行为推理：我们的水平如何？》](https://doi.org/10.1109/ICSE55347.2025.00087)（ICSE 2025） - **许可证**：MIT