SymPrompt Focal Method Benchmark for Unit Test Generation

Benchmark focal methods for our paper <i>Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM</i>, to appear in FSE 2024. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompts's approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement \approach using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.本内容对应我们即将发表于FSE 2024的论文《代码感知提示：基于大语言模型（Large Language Model）的回归场景下覆盖引导式测试生成研究》（Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM），旨在介绍该论文中的基准聚焦方法。在该论文中，我们提出了SymPrompt——一种面向大语言模型、用于测试生成的代码感知提示策略。SymPrompt的设计思路基于近期研究成果：现有研究表明，当通过提示引导大语言模型以多步推理的方式处理问题时，其能够解决更为复杂的逻辑问题。我们将该方法论应用于测试生成任务：将测试套件生成流程拆解为多阶段序列，每个阶段均由与待测方法执行路径相匹配的专属提示驱动，并向模型暴露相关类型与依赖关系的聚焦上下文。我们的方法可使预训练大语言模型无需额外训练即可生成更为完整的测试用例。我们基于TreeSitter解析框架实现了SymPrompt，并在来自开源Python项目的基准挑战性方法集上开展了评估。在CodeGen2模型上，SymPrompt可将正确测试用例生成量提升5倍，并将相对覆盖率提升26%。尤为值得注意的是，当将SymPrompt应用于GPT-4模型时，相较于基线提示策略，其覆盖率提升幅度超过2倍。