formulacode/formulacode-all

---
configs:
- config_name: "2017-10"
data_files:
- split: train
path: "2017-10/train-00000-of-00001.parquet"
- config_name: "2018-05"
data_files:
- split: train
path: "2018-05/train-00000-of-00001.parquet"
- config_name: "2018-07"
data_files:
- split: train
path: "2018-07/train-00000-of-00001.parquet"
- config_name: "2018-09"
data_files:
- split: train
path: "2018-09/train-00000-of-00001.parquet"
- config_name: "2018-11"
data_files:
- split: train
path: "2018-11/train-00000-of-00001.parquet"
- config_name: "2019-01"
data_files:
- split: train
path: "2019-01/train-00000-of-00001.parquet"
- config_name: "2019-06"
data_files:
- split: train
path: "2019-06/train-00000-of-00001.parquet"
- config_name: "2019-09"
data_files:
- split: train
path: "2019-09/train-00000-of-00001.parquet"
- config_name: "2020-01"
data_files:
- split: train
path: "2020-01/train-00000-of-00001.parquet"
- config_name: "2020-02"
data_files:
- split: train
path: "2020-02/train-00000-of-00001.parquet"
- config_name: "2020-03"
data_files:
- split: train
path: "2020-03/train-00000-of-00001.parquet"
- config_name: "2020-06"
data_files:
- split: train
path: "2020-06/train-00000-of-00001.parquet"
- config_name: "2020-07"
data_files:
- split: train
path: "2020-07/train-00000-of-00001.parquet"
- config_name: "2020-08"
data_files:
- split: train
path: "2020-08/train-00000-of-00001.parquet"
- config_name: "2020-09"
data_files:
- split: train
path: "2020-09/train-00000-of-00001.parquet"
- config_name: "2020-10"
data_files:
- split: train
path: "2020-10/train-00000-of-00001.parquet"
- config_name: "2020-11"
data_files:
- split: train
path: "2020-11/train-00000-of-00001.parquet"
- config_name: "2021-01"
data_files:
- split: train
path: "2021-01/train-00000-of-00001.parquet"
- config_name: "2021-02"
data_files:
- split: train
path: "2021-02/train-00000-of-00001.parquet"
- config_name: "2021-03"
data_files:
- split: train
path: "2021-03/train-00000-of-00001.parquet"
- config_name: "2021-04"
data_files:
- split: train
path: "2021-04/train-00000-of-00001.parquet"
- config_name: "2021-05"
data_files:
- split: train
path: "2021-05/train-00000-of-00001.parquet"
- config_name: "2021-06"
data_files:
- split: train
path: "2021-06/train-00000-of-00001.parquet"
- config_name: "2021-07"
data_files:
- split: train
path: "2021-07/train-00000-of-00001.parquet"
- config_name: "2021-08"
data_files:
- split: train
path: "2021-08/train-00000-of-00001.parquet"
- config_name: "2021-09"
data_files:
- split: train
path: "2021-09/train-00000-of-00001.parquet"
- config_name: "2021-10"
data_files:
- split: train
path: "2021-10/train-00000-of-00001.parquet"
- config_name: "2021-11"
data_files:
- split: train
path: "2021-11/train-00000-of-00001.parquet"
- config_name: "2021-12"
data_files:
- split: train
path: "2021-12/train-00000-of-00001.parquet"
- config_name: "2022-01"
data_files:
- split: train
path: "2022-01/train-00000-of-00001.parquet"
- config_name: "2022-02"
data_files:
- split: train
path: "2022-02/train-00000-of-00001.parquet"
- config_name: "2022-03"
data_files:
- split: train
path: "2022-03/train-00000-of-00001.parquet"
- config_name: "2022-04"
data_files:
- split: train
path: "2022-04/train-00000-of-00001.parquet"
- config_name: "2022-05"
data_files:
- split: train
path: "2022-05/train-00000-of-00001.parquet"
- config_name: "2022-06"
data_files:
- split: train
path: "2022-06/train-00000-of-00001.parquet"
- config_name: "2022-07"
data_files:
- split: train
path: "2022-07/train-00000-of-00001.parquet"
- config_name: "2022-08"
data_files:
- split: train
path: "2022-08/train-00000-of-00001.parquet"
- config_name: "2022-09"
data_files:
- split: train
path: "2022-09/train-00000-of-00001.parquet"
- config_name: "2022-10"
data_files:
- split: train
path: "2022-10/train-00000-of-00001.parquet"
- config_name: "2022-11"
data_files:
- split: train
path: "2022-11/train-00000-of-00001.parquet"
- config_name: "2022-12"
data_files:
- split: train
path: "2022-12/train-00000-of-00001.parquet"
- config_name: "2023-01"
data_files:
- split: train
path: "2023-01/train-00000-of-00001.parquet"
- config_name: "2023-02"
data_files:
- split: train
path: "2023-02/train-00000-of-00001.parquet"
- config_name: "2023-03"
data_files:
- split: train
path: "2023-03/train-00000-of-00001.parquet"
- config_name: "2023-04"
data_files:
- split: train
path: "2023-04/train-00000-of-00001.parquet"
- config_name: "2023-05"
data_files:
- split: train
path: "2023-05/train-00000-of-00001.parquet"
- config_name: "2023-06"
data_files:
- split: train
path: "2023-06/train-00000-of-00001.parquet"
- config_name: "2023-07"
data_files:
- split: train
path: "2023-07/train-00000-of-00001.parquet"
- config_name: "2023-08"
data_files:
- split: train
path: "2023-08/train-00000-of-00001.parquet"
- config_name: "2023-09"
data_files:
- split: train
path: "2023-09/train-00000-of-00001.parquet"
- config_name: "2023-10"
data_files:
- split: train
path: "2023-10/train-00000-of-00001.parquet"
- config_name: "2023-11"
data_files:
- split: train
path: "2023-11/train-00000-of-00001.parquet"
- config_name: "2023-12"
data_files:
- split: train
path: "2023-12/train-00000-of-00001.parquet"
- config_name: "2024-01"
data_files:
- split: train
path: "2024-01/train-00000-of-00001.parquet"
- config_name: "2024-02"
data_files:
- split: train
path: "2024-02/train-00000-of-00001.parquet"
- config_name: "2024-03"
data_files:
- split: train
path: "2024-03/train-00000-of-00001.parquet"
- config_name: "2024-04"
data_files:
- split: train
path: "2024-04/train-00000-of-00001.parquet"
- config_name: "2024-05"
data_files:
- split: train
path: "2024-05/train-00000-of-00001.parquet"
- config_name: "2024-06"
data_files:
- split: train
path: "2024-06/train-00000-of-00001.parquet"
- config_name: "2024-07"
data_files:
- split: train
path: "2024-07/train-00000-of-00001.parquet"
- config_name: "2024-08"
data_files:
- split: train
path: "2024-08/train-00000-of-00001.parquet"
- config_name: "2024-09"
data_files:
- split: train
path: "2024-09/train-00000-of-00001.parquet"
- config_name: "2024-10"
data_files:
- split: train
path: "2024-10/train-00000-of-00001.parquet"
- config_name: "2024-11"
data_files:
- split: train
path: "2024-11/train-00000-of-00001.parquet"
- config_name: "2024-12"
data_files:
- split: train
path: "2024-12/train-00000-of-00001.parquet"
- config_name: "2025-01"
data_files:
- split: train
path: "2025-01/train-00000-of-00001.parquet"
- config_name: "2025-02"
data_files:
- split: train
path: "2025-02/train-00000-of-00001.parquet"
- config_name: "2025-03"
data_files:
- split: train
path: "2025-03/train-00000-of-00001.parquet"
- config_name: "2025-04"
data_files:
- split: train
path: "2025-04/train-00000-of-00001.parquet"
- config_name: "2025-05"
data_files:
- split: train
path: "2025-05/train-00000-of-00001.parquet"
- config_name: "2025-06"
data_files:
- split: train
path: "2025-06/train-00000-of-00001.parquet"
- config_name: "2025-07"
data_files:
- split: train
path: "2025-07/train-00000-of-00001.parquet"
- config_name: "2025-08"
data_files:
- split: train
path: "2025-08/train-00000-of-00001.parquet"
- config_name: "2025-09"
data_files:
- split: train
path: "2025-09/train-00000-of-00001.parquet"
- config_name: "2025-10"
data_files:
- split: train
path: "2025-10/train-00000-of-00001.parquet"
- config_name: "2025-11"
data_files:
- split: train
path: "2025-11/train-00000-of-00001.parquet"
- config_name: "default"
data_files:
- split: train
path: "default/train-00000-of-00001.parquet"
- config_name: "verified"
data_files:
- split: train
path: "verified/train-00000-of-00001.parquet"
default_config_name: "default"
task_categories:
- text-generation
tags:
- code
- performance-optimization
- benchmark
language:
- en
size_categories:
- 1K<n<10K
---


<p align="center">
<a href="https://formula-code.github.io/">
<img src="https://img.shields.io/badge/%F0%9F%8C%90%20Website-0A7A5E?style=for-the-badge" alt="FormulaCode Website">
</a>
<a href="https://huggingface.co/papers/2603.16011">
<img src="https://img.shields.io/badge/Paper-1F6FEB?style=for-the-badge&logo=arxiv&logoColor=white" alt="FormulaCode Paper">
</a>
<a href="https://formula-code.github.io/leaderboard/">
<img src="https://img.shields.io/badge/%F0%9F%93%88%20Leaderboard-EA580C?style=for-the-badge&logoColor=white" alt="FormulaCode Leaderboard">
</a>
</p>

[FormulaCode](https://formula-code.github.io/) is a *live* benchmark for evaluating the holistic ability of LLM agents to optimize codebases. FormulaCode consists of two parts: a [pipeline](https://github.com/formula-code/datasmith) to construct performance optimization tasks, and an [execution harness](https://github.com/formula-code/terminal-bench) that connects a language model to our terminal sandbox.

This dataset contains **1215** enriched performance optimization tasks derived from real open-source Python projects, spanning 76 months of merged PRs.

The dataset is continuously updated with new tasks every month (available under `YYYY-MM` configs, and `default` for all tasks), and a subset of tasks are human-validated and labeled as `verified`.

## Quick Start

```python
from datasets import load_dataset

# Load all tasks
ds = load_dataset("formulacode/formulacode-all")

# Load only verified tasks (human-validated)
ds = load_dataset("formulacode/formulacode-all", "verified")

# Load tasks from a specific month
ds = load_dataset("formulacode/formulacode-all", "2024-07")
```

## Configs

| Config | Description | Tasks |
|--------|-------------|-------|
| `default` | All tasks | 1215 |
| `verified` | Human-validated subset | 108 |
| `YYYY-MM` | Tasks by PR merge month (76 months available) | varies |

## Key Columns

| Column | Description |
|--------|-------------|
| `task_id` | Unique task identifier (e.g. `pandas-dev_pandas_1`) |
| `repo_name` | Source repository (e.g. `pandas-dev/pandas`) |
| `container_name` | Docker container reference (`<owner>-<repo>-<sha>:final`) |
| `image_name` | Full Docker Hub image reference |
| `difficulty` | Normalized difficulty: `easy`, `medium`, `hard` |
| `classification` | Optimization type (e.g. `use_better_algorithm`, `micro_optimizations`) |
| `patch` | Ground truth performance improvement patch |
| `final_md` | Task instructions in markdown |
| `pr_merged_at` | Date the PR was merged |
| `pr_merge_commit_sha` | Merge commit SHA |
| `pr_base_sha` | Base commit SHA |

配置项：
- 配置名称: "2017-10"
数据文件:
- 拆分集: 训练集
路径: "2017-10/train-00000-of-00001.parquet"
- 配置名称: "2018-05"
数据文件:
- 拆分集: 训练集
路径: "2018-05/train-00000-of-00001.parquet"
（其余按月度划分的配置项格式一致，覆盖2017年10月至2025年11月的所有月份，共计76个配置项）
- 配置名称: "default"
数据文件:
- 拆分集: 训练集
路径: "default/train-00000-of-00001.parquet"
- 配置名称: "verified"
数据文件:
- 拆分集: 训练集
路径: "verified/train-00000-of-00001.parquet"
默认配置名称: "default"
任务类别:
- 文本生成
标签:
- 代码
- 性能优化
- 基准测试
语言:
- 英语
规模类别:
- 1000 < 样本量 < 10000


<p align="center">
<a href="https://formula-code.github.io/">
<img src="https://img.shields.io/badge/%F0%9F%8C%90%20%E5%AE%98%E7%BD%91-0A7A5E?style=for-the-badge" alt="FormulaCode%20%E5%AE%98%E7%BD%91">
</a>
<a href="https://huggingface.co/papers/2603.16011">
<img src="https://img.shields.io/badge/%E8%AE%BA%E6%96%87-1F6FEB?style=for-the-badge&logo=arxiv&logoColor=white" alt="FormulaCode%20%E8%AE%BA%E6%96%87">
</a>
<a href="https://formula-code.github.io/leaderboard/">
<img src="https://img.shields.io/badge/%E6%8E%92%E8%A1%A8%E6%9D%BF-EA580C?style=for-the-badge&logoColor=white" alt="FormulaCode%20%E6%8E%92%E8%A1%A8%E6%9D%BF">
</a>
</p>

[FormulaCode](https://formula-code.github.io/) 是一款用于评估大语言模型智能体（LLM Agent）优化代码库综合能力的**实时基准测试集**。该基准测试集包含两部分：一是用于构建性能优化任务的流水线（pipeline），二是将大语言模型连接至终端沙箱的执行测试套件（execution harness）。

本数据集包含1215个源自真实开源Python项目的增强型性能优化任务，涵盖76个月的合并拉取请求（Pull Request，PR）。

该数据集每月持续更新新增任务，可通过`YYYY-MM`格式的配置加载对应月份的任务，`default`配置用于加载全部任务；其中部分任务经过人工验证，被标记为`verified`（已验证任务）。

## 快速入门

python
from datasets import load_dataset

# 加载全部任务
ds = load_dataset("formulacode/formulacode-all")

# 仅加载已验证的人工校验任务
ds = load_dataset("formulacode/formulacode-all", "verified")

# 加载指定月份的任务
ds = load_dataset("formulacode/formulacode-all", "2024-07")

## 配置说明

| 配置名称 | 配置说明 | 任务数量 |
|--------|-------------|-------|
| `default` | 全部任务 | 1215 |
| `verified` | 人工验证子集 | 108 |
| `YYYY-MM` | 按PR合并月份划分的任务（共76个可选配置） | 数量不定 |

## 关键字段

| 字段名 | 字段说明 |
|--------|-------------|
| `task_id` | 唯一任务标识符（例如`pandas-dev_pandas_1`） |
| `repo_name` | 源代码仓库（例如`pandas-dev/pandas`） |
| `container_name` | Docker容器引用（格式为`<owner>-<repo>-<sha>:final`） |
| `image_name` | 完整的Docker Hub镜像引用 |
| `difficulty` | 归一化难度等级：`easy`（简单）、`medium`（中等）、`hard`（困难） |
| `classification` | 优化类型（例如`use_better_algorithm`（使用更优算法）、`micro_optimizations`（微优化）） |
| `patch` | 真实性能优化补丁 |
| `final_md` | Markdown格式的任务指令 |
| `pr_merged_at` | PR合并日期 |
| `pr_merge_commit_sha` | 合并提交的SHA哈希值 |
| `pr_base_sha` | 基础提交SHA哈希值 |