HumanEvalComm

HumanEvalComm是由不列颠哥伦比亚大学创建的数据集，包含164个手工修改的Python编程问题，旨在评估大型语言模型（LLMs）在代码生成任务中的沟通能力。数据集通过引入模糊性、不一致性和不完整性等要求工程概念，促使模型在生成代码前提出澄清问题。创建过程涉及手动修改原始问题描述，确保每个问题都能触发澄清问题，以提高代码生成的准确性。该数据集主要应用于评估和提升LLMs在软件工程领域的沟通技能，特别是在处理复杂或不明确的需求时。HumanEvalComm is a dataset developed by the University of British Columbia, consisting of 164 hand-modified Python programming problems. It is designed to evaluate the communication capabilities of Large Language Models (LLMs) in code generation tasks. By introducing requirements engineering concepts such as ambiguity, inconsistency, and incompleteness, the dataset prompts models to raise clarification questions prior to code generation. The dataset creation process entails manually revising original problem descriptions to ensure that each problem can elicit clarification questions, thus enhancing the accuracy of code generation. This dataset is primarily applied to evaluate and improve the communication skills of LLMs in the software engineering domain, particularly when dealing with complex or ambiguous requirements.