humanevalpack

HumanEvalPack旨在扩展OpenAI的HumanEval，覆盖6种编程语言的3种任务场景。它提供Python、JavaScript、Java、Go、C++和Rust等多种语言的数据，其中Python部分与OpenAI的Python HumanEval完全一致。数据集中包含任务ID、提示词、函数声明、标准解答、带缺陷的解答及其缺陷类型、入口点、必要的导入信息、单元测试、函数签名、文档字符串以及指令等字段。此数据集可用于代码生成模型的指令调优和评估，并提供标准化数据操作。数据集基于MIT许可发布。

HumanEvalPack is designed to extend OpenAI's HumanEval, covering 3 task scenarios across 6 programming languages. It provides datasets in multiple programming languages including Python, JavaScript, Java, Go, C++, and Rust, where the Python subset is entirely consistent with OpenAI's Python HumanEval. The dataset includes fields such as task ID, prompt, function declaration, canonical solution, buggy solution and its bug type, entry point, required imports, unit tests, function signature, docstring, and instructions. This dataset can be used for instruction tuning and evaluation of code generation models, and supports standardized data manipulation. The dataset is released under the MIT License.

搜集汇总

背景与挑战

背景概述

HumanEvalPack是一个多语言代码评估数据集，扩展自OpenAI HumanEval，覆盖6种主流编程语言和3种任务场景。该数据集提供完整的代码评估要素(包括标准解答、带缺陷解答和测试用例)，主要用于代码生成模型的指令调优和性能评估。