SpringProd and ApacheProd - executable text-code datasets

M. Kacmajor and J.D. Kelleher, "ExTra: Evaluation of Automatically Generated Source Code Using Execution Traces" (submitted to IEEE TSE)In this paper we propose ExTra---a novel approach to evaluating code quality based on the comparison of execution traces of the generated code and the ground-truth code. ExTra captures the behaviour of the programs implemented with the generated code, taking into account all the internal and external dependencies. In contrast to source-code based metrics, ExTra is semantically meaningful; and in contrast to the evaluation approaches measuring the functional correctness of code, ExTra is suitable for evaluation of code developed in the context of real-life software systems. The first contribution of this paper is the design, implementation, and validation of ExTra. The value of ExTra is examined via experiments in which our metric and three source-code based metrics (BLEU, Levenshtein distance and CodeBLEU) are applied to two types of automatically generated source code: test code and production code. The results show that the scores produced by the three source-code based metrics are highly correlated, while ExTra is clearly distinct. The qualitative analysis of the differences reveals a number of examples of ExTra scores being semantically more adequate than the scores computed based on token comparison. Furthermore, the quantitative analysis of the agreement between the evaluation scores and test verdicts---produced by generated test cases or by test cases applied to the generated code---shows that ExTra is a much better predictor of verdicts \textit{failed} than any of the three text-oriented metrics. On the whole, our results indicate that ExTra provides added value to the process of assessing the quality of the generated code, and we recommend it as an evaluation tool complementary to the source-code based methods. The second contribution of this paper are three new evaluation datasets which contain executable code extracted from large, active Github repositories and can be used for evaluting models' performance using ExTra, or for other tasks that require executable code.

M. Kacmajor 和 J.D. Kelleher，"ExTra：基于执行轨迹自动生成源代码质量评估方法"（提交至IEEE TSE）本文提出了ExTra——一种新颖的代码质量评估方法，该方法通过比较自动生成代码与真实代码的执行轨迹来进行。ExTra捕捉使用生成代码实现的程序的行为，考虑到所有内部和外部依赖。与基于源代码的指标不同，ExTra具有语义上的意义；与衡量代码功能正确性的评估方法相比，ExTra更适合于评估现实软件系统中的代码。本文的第一项贡献是ExTra的设计、实现和验证。通过实验检验了ExTra的价值，其中我们的指标和三种基于源代码的指标（BLEU、Levenshtein距离和CodeBLEU）应用于两种类型的自动生成源代码：测试代码和生成代码。结果表明，三种基于源代码的指标产生的分数高度相关，而ExTra则明显不同。对差异的定性分析揭示了ExTra分数在语义上比基于Token比较计算出的分数更为恰当的多个示例。此外，对评估分数与测试结果之间的一致性的定量分析——由生成测试用例或应用于生成代码的测试用例产生的测试结果——表明，ExTra是比任何三种基于文本的指标都更能预测结果"失败"的。总体而言，我们的结果表明，ExTra为评估生成代码的质量提供了额外的价值，我们建议将其作为源代码方法的一种补充评估工具。本文的第二项贡献是三个新的评估数据集，其中包含从大型、活跃的GitHub仓库中提取的可执行代码，可用于使用ExTra评估模型性能，或用于需要可执行代码的其他任务。