ernie-research/MEnvBench

---
language:
- en
license: apache-2.0
size_categories:
- 1K<n<10K
task_categories:
- text-generation
pretty_name: MEnvBench
tags:
- code
- software-engineering
- benchmark
- environment-construction
- multi-language
---

# MEnvBench: Multi-Language Environment Construction Benchmark

<p align="center">
<a href="https://arxiv.org/abs/2601.22859"><img src="https://img.shields.io/badge/arXiv-2601.22859-b31b1b.svg"></a>
<a href="https://github.com/ernie-research/MEnvAgent"><img src="https://img.shields.io/badge/GitHub-MEnvAgent-blue"></a>
<a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-green.svg"></a>
</p>

<p align="center">
<a href="https://huggingface.co/datasets/ernie-research/MEnvData-SWE"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HF-MEnvData--SWE-orange"></a>
<a href="https://huggingface.co/datasets/ernie-research/MEnvData-SWE-Trajectory"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HF-MEnvData--SWE--Trajectory-red"></a>
</p>

## 📋 Dataset Description

**MEnvBench** is a comprehensive benchmark for evaluating multi-language environment building and test execution capabilities, comprising **1,000 task instances** (10 languages × 20 repositories × 5 instances) selected from 200 high-quality open-source repositories.

### Key Features

- 🌐 **Multi-Language Coverage**: 10 programming languages (Python, Java, TypeScript, JavaScript, Rust, Go, C++, Ruby, PHP, C)
- 🎯 **High Quality**: Multi-stage filtering pipeline from 8,000 candidate repositories (>1,000 stars, >200 forks/issues/PRs)
- 📊 **Diverse Domains**: Strategic sampling across AI, System, Web, and other software ecosystems
- 🏗️ **Difficulty Levels**: Five project scale bands from <10MB to >500MB
- ✅ **Verified Quality**: Closed issues with test patches and LLM-based quality assessment

### Dataset Statistics

| Metric | Value |
|--------|-------|
| **Total Instances** | 1,000 |
| **Languages** | 10 |
| **Repositories** | 200 |
| **Instge** | 100 (20 repos × 5 instances) |

## 📊 Dataset Structure

Each instance in MEnvBench contains the following fields:

| Field | Type | Description |
| :--- | :--- | :--- |
| **repo** | `str` | The full GitHub repository name (e.g., "home-assistant/core"). |
| **pull_number** | `int` | The pull request number associated with the fix (e.g., 807). |
| **instance_id** | `str` | A unique identifier for the task instance (e.g., "home-assistant__core-807"). |
| **issue_numbers** | `list` | A list of linked issue numbers (e.g., [103876]). |
| **base_commit** | `str` | The commit SHA of the repository prior to the fix. |
| **version** | `str` | The version of the dataset (e.g., "0.10"). |
| **patch** | `str` | The ground-truth patch (git diff) that resolves the issue. |
| **test_patch** | `str` | The test patch (git diff) containing new tests to reproduce the issue. |
| **problem_statement** | `str` | The natural language description of the issue. |
| **hints_text** | `str` | Hints extracted from the issue dussion to aid resolution. |
| **all_hints_text** | `str` | Comprehensive context including all comments and code review details. |
| **commit_urls** | `list` | A list of URLs pointing to the relevant commits. |
| **created_at** | `str` | The creation timestamp (e.g., "2015-12-27T19:33:55Z"). |
| **language** | `str` | The programming language of the repository (e.g., "Python"). |

## 🚀 Usage

### Loading the Dataset

```python
from datasets import load_dataset

# Load the full dataset
dataset = load_dataset("ernie-research/MEnvBench")

# Access a single instance
instance = dataset['test'][0]
print(f"Repository: {instance['repo']}")
print(f"Language: {instance['language']}")
print(f"Problem: {instance['problem_statement'][:200]}...")
```

### Example Instance

```python
{
"repo": "home-assistant/core",
"pull_number": 807,
"instance_id": "home-assistant__core-807",
"issue_numbers": [103876],
"base_commit": "87c88078c87257cde4786997fedb865be6813545",
"version": "0.10",
"language""Python",
"problem_statement": "Scene configuration issue...",
"patch": "diff --git a/homeassistant/components/scene.py...",
"test_patch": "diff --git a/tests/components/test_scene.py...",
...
}
```

## 📖 Citation

If MEnvBench helps your research, please cite:

```bibtex
@misc{guo2026menvagent,
title={MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering},
author={Chuanzhe Guo and Jingjing Wu and Sijun He and Yang Chen and Zhaoqi Kuang and Shilong Fan and Bingjin Chen and Siqi Bao and Jing Liu and Hua Wu and Qingfu Zhu and Wanxiang Che and Haifeng Wang},
year={2026},
url={https://arxiv.org/abs/2601.22859},
}
```

## 📧 Contact

For questions or issues:
- **GitHub**: [MEnvAgent Repository](https://github.com/ernie-research/MEnvAgent)
- **Email**: [e-mail]

## 🙏 Acknowledgements

We thank the open-source community and all repository maintainers whose high-quality projects made this benchmark possible.

### 数据集元数据
- 语言：英语
- 许可证：Apache 2.0
- 样本规模：1000 < 样本数量 < 10000
- 任务类别：文本生成
- 展示名称：MEnvBench
- 标签：代码、软件工程、基准测试、环境构建、多语言

# MEnvBench：多语言环境构建基准测试

<p align="center">
<a href="https://arxiv.org/abs/2601.22859"><img src="https://img.shields.io/badge/arXiv-2601.22859-b31b1b.svg"></a>
<a href="https://github.com/ernie-research/MEnvAgent"><img src="https://img.shields.io/badge/GitHub-MEnvAgent-blue"></a>
<a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache_2.0-green.svg"></a>
</p>

<p align="center">
<a href="https://huggingface.co/datasets/ernie-research/MEnvData-SWE"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HF-MEnvData--SWE-orange"></a>
<a href="https://huggingface.co/datasets/ernie-research/MEnvData-SWE-Trajectory"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20HF-MEnvData--SWE--Trajectory-red"></a>
</p>

## 📋 数据集说明

**MEnvBench**是一款用于评估多语言环境构建与测试执行能力的综合基准测试集，包含**1000个任务实例**（覆盖10种编程语言 × 20个仓库 × 5个实例），样本源自200个高质量开源仓库。

### 核心特性

- 🌐 **多语言覆盖**：涵盖10种编程语言（Python、Java、TypeScript、JavaScript、Rust、Go、C++、Ruby、PHP、C）
- 🎯 **高质量筛选**：从8000个候选仓库（星标数>1000，复刻/议题/拉取请求数>200）中通过多阶段过滤流程构建
- 📊 **领域多样性**：覆盖人工智能、系统开发、Web开发等多个软件生态的战略抽样样本
- 🏗️ **难度梯度**：包含5个项目规模层级，从小于10MB到大于500MB不等
- ✅ **质量验证**：带有已关闭议题的测试补丁以及基于大语言模型（LLM）的质量评估

### 数据集统计数据

| 指标 | 数值 |
|--------|-------|
| **总任务实例数** | 1000 |
| **覆盖语言数** | 10 |
| **关联仓库数** | 200 |
| **Instge** | 100（20个仓库 × 5个实例）|

## 📊 数据集结构

MEnvBench中的每个任务实例包含以下字段：

| 字段 | 类型 | 描述 |
| :--- | :--- | :--- |
| **repo** | `字符串` | GitHub仓库的完整名称（例如："home-assistant/core"）。 |
| **pull_number** | `整数` | 对应修复操作的拉取请求编号（例如：807）。 |
| **instance_id** | `字符串` | 任务实例的唯一标识符（例如："home-assistant__core-807"）。 |
| **issue_numbers** | `列表` | 关联议题编号的列表（例如：[103876]）。 |
| **base_commit** | `字符串` | 修复前的仓库提交SHA哈希值。 |
| **version** | `字符串` | 数据集版本（例如："0.10"）。 |
| **patch** | `字符串` | 用于解决该议题的真实补丁（git diff格式）。 |
| **test_patch** | `字符串` | 用于复现议题的测试补丁（git diff格式，包含新增测试用例）。 |
| **problem_statement** | `字符串` | 议题的自然语言描述。 |
| **hints_text** | `字符串` | 从议题讨论中提取的、用于辅助解决问题的提示信息。 |
| **all_hints_text** | `字符串` | 包含所有评论与代码审查细节的完整上下文信息。 |
| **commit_urls** | `列表` | 指向相关提交的URL列表。 |
| **created_at** | `字符串` | 数据集创建时间戳（例如："2015-12-27T19:33:55Z"）。 |
| **language** | `字符串` | 仓库使用的编程语言（例如："Python"）。 |

## 🚀 使用指南

### 加载数据集

python
from datasets import load_dataset

# 加载完整数据集
dataset = load_dataset("ernie-research/MEnvBench")

# 访问单个实例
instance = dataset['test'][0]
print(f"仓库名称：{instance['repo']}")
print(f"编程语言：{instance['language']}")
print(f"问题描述：{instance['problem_statement'][:200]}...")

### 实例示例

python
{
"repo": "home-assistant/core",
"pull_number": 807,
"instance_id": "home-assistant__core-807",
"issue_numbers": [103876],
"base_commit": "87c88078c87257cde4786997fedb865be6813545",
"version": "0.10",
"language": "Python",
"problem_statement": "场景配置问题...",
"patch": "diff --git a/homeassistant/components/scene.py...",
"test_patch": "diff --git a/tests/components/test_scene.py...",
...
}

## 📖 引用格式

如果MEnvBench对您的研究有所帮助，请引用以下文献：

bibtex
@misc{guo2026menvagent,
title={MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering},
author={Chuanzhe Guo and Jingjing Wu and Sijun He and Yang Chen and Zhaoqi Kuang and Shilong Fan and Bingjin Chen and Siqi Bao and Jing Liu and Hua Wu and Qingfu Zhu and Wanxiang Che and Haifeng Wang},
year={2026},
url={https://arxiv.org/abs/2601.22859},
}

## 📧 联系方式

如有疑问或问题，请联系：
- **GitHub仓库**：[MEnvAgent 仓库](https://github.com/ernie-research/MEnvAgent)
- **邮箱**：[e-mail]

## 🙏 致谢

感谢开源社区与所有仓库维护者贡献的高质量项目，为本基准测试集的构建提供了基础。