SymPrompt Focal Method Benchmark for Unit Test Generation

Benchmark focal methods for our paper <i>Code-Aware Prompting: A Study of Coverage-Guided Test Generation in Regression Setting using LLM</i>, to appear in FSE 2024. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompts's approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement \approach using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.