A Collection of Real-World Benchmark Tasks for Repair of Partial Program Fixes

Debugging is a time-consuming process in software development. Even if a programmer proposes a fix it is not guaranteed that it eliminates the problem completely. Such fixes are called partial fixes. However, in recent years, the popularity of formal verification and automated program repair tools has grown. They assist in locating and eliminating the bug. Automated repair tools additionally propose a bug-fix. In this work, we describe a way to mine a benchmark set for these tools to help them recognize partial fixes and suggesting the full fix instead. Our approach uses the GitHub API to fetch closed issues as candidates for partial fixes. We combine two strategies to filter and score the most interesting issues. The first strategy searches for reopened issues as they indicate that a previous commit has not fixed the program completely. The second strategy considers the CI status. If at least one but the last commit has a failing CI, we consider it as a candidate for a partial fix. Many of the resulting top-rated issues represent actual partial fixes. Although the number of filtered real partial fixes on GitHub is small compared to the number of candidates, we are able to present a growing benchmark set of over 2,000 partial fixes.