Manual Inspection Dataset of 145 Checkstyle Issues for Evaluating the Issue-Selector

This Zenodo record provides a supplementary spreadsheet containing the manual human inspection of 145 GitHub issues from the Checkstyle project. The artifact complements the evaluation of the Issue-Selector presented in the paper “Detecting, Ranking, and Planning Issue-Driven Refactorings with a Multi-Agent Architecture,” published at SBES 2026. For each issue, the spreadsheet reports its experimental group, GitHub identifier and URL, title, the classification produced by the Issue-Selector, the human-estimated distribution across the refactor, feature, other, and bug categories, the final human judgment—Match, Borderline, or Wrong—and a textual justification. The judgments follow a formal rubric that considers whether the dominant issue kind is correct, whether all included issue kinds are supported by textual evidence, and whether relevant issue kinds were omitted. Match represents a fully supported classification, Borderline represents a directionally correct but debatable classification, and Wrong represents a classification containing a confirmed error. The workbook contains four worksheets: Rubric: definitions and decision criteria used during the manual inspection; Judgment: issue-level human classifications and justifications; Comparison: comparison between the manual judgments and the previous LLM-assisted judgments; Summary: aggregated results by experimental group, including the number of Match, Borderline, and Wrong classifications and the directional accuracy, calculated as (Match + Borderline) / Total. This supplementary artifact is provided to improve the transparency, traceability, and reproducibility of the Issue-Selector evaluation. It should be used together with the main replication package available at DOI: 10.5281/zenodo.20031732.