palaestraresearch/bigcodebench-plus

BCBPlus（BigCodeBench-Plus）是一个经过修复和改进的代码生成基准测试数据集，专门用于Python语言。它修复了原始数据集中的规范模糊性、测试错误和损坏的规范解决方案。数据集包含1136个任务，每个任务包括完整的提示、指令提示、代码提示、规范解决方案、测试代码、入口点和使用的库。数据集的构建哲学强调确定性文档字符串示例作为规范，遵循库的约定，函数签名的约束性，不进行精确的RNG实现测试，以及不测试错误消息的具体措辞。数据集可用于代码生成任务的研究和评估。

BCBPlus (BigCodeBench-Plus) is a fixed and improved code generation benchmark dataset specifically for Python. It corrects specification ambiguities, test bugs, and broken canonical solutions from the original dataset. The dataset contains 1136 tasks, each including a complete prompt, instruction prompt, code prompt, canonical solution, test code, entry point, and libraries used. The curation philosophy emphasizes deterministic docstring examples as specifications, adherence to library conventions, binding function signatures, no exact RNG-realization tests, and no error-message wording tests. The dataset is suitable for research and evaluation in code generation tasks.