Exploring Fine-Grained Bug Report Categorization with Large Language Models and Prompt Engineering: An Empirical Study

Replication Package for: Exploring Fine-Grained Bug Report Categorization with Large Language Models and Prompt Engineering: An Empirical Study This repository contains the scripts to replicate the experiments for the paper "Exploring Fine-Grained Bug Report Categorization with Large Language Models and Prompt Engineering: An Empirical Study." Zenodo The replication package is available at Zenodo. Dependencies The following dependencies are required to run the experiments: Ollama: The user must download models from Ollama before executing the pipeline. Anaconda / Miniconda: Download and configure Anaconda or Miniconda for managing Python environments. Create a Python environment using the environment file: conda env create -f environment.yml How to Run The main script to run the experiments is prompt.py. To execute the experiment: conda activate llm-fine-grained-bug-categorization python prompt.py -c config.ini config.ini contains the configurations for the experiments. Below is an example of the configuration file: [Model] name = gemma2:9b-instruct-q4_0 temperature = 0 set_system = True num_ctx = 8192 num_runs = 1 ;diid = XERCESC-211 ;host = http://localhost:11435 [Prompt] reverse = True type = 83215003 Configuration Explanation: name: Specifies the model name. You can list multiple models separated by commas. name = llama3.1:8b-instruct-q4_0, llama3:8b-instruct-q4_0, qwen2:7b-instruct-q4_0, gemma:7b-instruct-v1.1-q4_0, gemma2:9b-instruct-q4_0, starling-lm:7b-beta-q4_0, aya:8b-23-q4_0 temperature: Defines the temperature for the LLM. set_system: Flag to set the system. Used to invoke the generate endpoint by setting the system message. num_ctx: LLM context size. num_runs: The number of runs to perform (currently only one run is supported). diid: Bug report ID (for debugging purposes, used to test the prompt and LLM). host: Host to use. reverse: Flag to reverse the execution order of the dataset. type: Specifies the prompt type. The number corresponds to the prompt template in the prompts/ folder. Project Structure |--- README.md : User guidance. |--- analysis : Folder to save the extracted categories and classification metrics. |--- base : Folder containing Ollama API implementation to interact with the LLM. |--- corrected_dataset : Folder containing the corrected dataset of bug report categories, refined through validation methods to improve human annotations. |--- experimentResults : Folder to save the LLM responses. |--- export : Folder containing the dataset of 221,184 LLM-generated bug report categories, created using six LLMs, nine prompt types, and four output configurations for 1,024 bug reports exported in HTML format. |--- prompts : Folder with prompt templates. |--- reports : Folder containing the bug report XML files (zipped with `.gz`). |--- all_non_sensical.xlsx : Contains all LLM responses for which no valid bug report category could be extracted. |--- all_non_sensical-odd-analysis.xlsx : Manual analysis of Out-of-Dictionary bug report categories generated by LLM. |--- dataset_additional_analysis.xlsx : Catolino dataset analysis. |--- config.ini : Example configuration file. Analysis Scripts The following scripts are used to analyze the results of the experiments. You can run them to replicate various parts of the analysis: analyze.py: Contains the script that calculates classification metrics for bug report categorization. analyzeCore.py: Contains the script for classification metrics analysis of bug report categorization. gainLoss.py: Analyzes the gain and loss of unique categories across different models and configurations. plotDiscardDist.py: Analyzes LLM responses and generates a plot of the distribution of common output deviations. labelCorrectness.py: Analyzes the correctness of labels using agreements and disagreements from LLM-generated responses via voting.