PythonSaga

PythonSaga是由印度理工学院甘地分校计算机科学与工程系Lingo研究组创建的一个新型代码生成基准数据集，包含185个手工制作的提示，涵盖38个编程概念，并平衡分布在不同难度级别。该数据集旨在解决现有基准在编程概念多样性和难度级别上的不足，通过提供一个更全面和平衡的评估框架，以更准确地评估大型语言模型在代码生成任务上的性能。数据集内容丰富，包括从基础到高级的编程概念，以及相应的难度级别划分，适用于评估和提升模型在各种编程任务上的表现。

PythonSaga is a novel code generation benchmark dataset created by the Lingo Research Group, Department of Computer Science and Engineering, Indian Institute of Technology Gandhinagar. It includes 185 handcrafted prompts covering 38 programming concepts, with a balanced distribution across various difficulty levels. This dataset aims to address the limitations of existing benchmarks in terms of programming concept diversity and difficulty level configuration, by providing a more comprehensive and balanced evaluation framework to accurately assess the performance of Large Language Models (LLMs) on code generation tasks. The dataset contains rich content ranging from basic to advanced programming concepts paired with corresponding difficulty level classifications, making it suitable for evaluating and improving model performance across diverse programming tasks.