HumanEval

这是论文“Evaluating Large Language Models Trained on Code”中描述的 HumanEval 问题解决数据集的评估工具。它用于测量从文档字符串合成程序的功能正确性。它由 164 个原始编程问题组成，评估语言理解、算法和简单的数学，还有一些
类似于简单的软件面试问题。

This is the evaluation toolkit for the HumanEval problem-solving dataset described in the paper *Evaluating Large Language Models Trained on Code*. It is designed to measure the functional correctness of programs synthesized from docstrings. It comprises 164 original programming problems that assess language comprehension, algorithmic proficiency, and basic mathematical reasoning, with some items similar to entry-level software engineering interview questions.

搜集汇总

背景与挑战

背景概述

HumanEval是一个包含164个编程问题的数据集，用于评估代码生成和程序合成的功能正确性，覆盖语言理解、算法和简单数学等领域。该数据集由OpenAI等机构于2021年发布，适用于文本预训练和代码生成研究。