Mercury

Mercury是由南洋理工大学和新加坡国立大学联合开发的第一个针对代码大型语言模型的计算效率基准数据集。该数据集包含1,889个Python任务，每个任务都有足够的解决方案来支持运行时分布。Mercury旨在评估和提高代码生成的计算效率，通过引入新的度量标准Beyond，该度量标准计算运行时百分位加权的通过分数，以同时反映功能正确性和计算效率。数据集的应用领域主要集中在提高代码生成模型的效率，确保生成的代码不仅正确而且高效。

Mercury is the first computational efficiency benchmark dataset for code large language models, jointly developed by Nanyang Technological University (NTU) and National University of Singapore (NUS). This dataset includes 1,889 Python tasks, each with sufficient valid solutions to support runtime distribution studies. Mercury aims to evaluate and enhance the computational efficiency of code generation by introducing a novel metric named Beyond, which computes a runtime percentile-weighted passing score to simultaneously reflect both functional correctness and computational efficiency. The primary applications of this dataset focus on improving the efficiency of code generation models, ensuring that the generated code is not only functionally correct but also computationally efficient.

搜集汇总

背景与挑战

背景概述

Mercury是由南洋理工大学和新加坡国立大学联合开发的代码大型语言模型计算效率基准数据集，包含1,889个Python任务，旨在评估和提高代码生成的计算效率。它引入Beyond度量标准，通过运行时百分位加权的通过分数，同时衡量功能正确性和计算效率，专注于提升代码生成模型的效率，确保代码既正确又高效。