Replication package for paper 'How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair'

Replication Package This replication package contains all data, annotations, and analysis scripts used in our study: How Do LLMs Read Bug Reports? An Empirical Study of Attention in LLMs for Automated Program Repair Structure ├── annotation/ │ ├── annotator1_java/ │ │ └── annotator1_annotations.json │ ├── annotator2_java/ │ │ └── annotator2_annotations.json │ ├── annotator3_python/ │ │ └── annotator3_annotations.json │ └── annotator4_python/ │ └── annotator4_annotations.json │ ├── pert_based_analysis/ │ ├── data/ │ │ ├── patches/ │ │ ├── Multi-SWE-Bench_Python_Qwen_attention_patterns.csv │ │ ├── Multi-SWE-Bench_java_Claude4.csv │ │ ├── Multi-SWE-Bench_java_GPT_OSS.csv │ │ ├── Multi-SWE-Bench_java_Qwen.csv │ │ ├── Multi-SWE-Bench_java_final.csv │ │ ├── Multi-SWE-Bench_java_final_perturbations.csv │ │ ├── Multi-SWE-Bench_java_final_perturbations_qwen.csv │ │ ├── Multi-SWE-Bench_java_qwen_attention_patterns.csv │ │ ├── Multi-SWE-Bench_python_Claude4.csv │ │ ├── Multi-SWE-Bench_python_GPT_OSS.csv │ │ ├── Multi-SWE-Bench_python_Qwen.csv │ │ ├── Multi-SWE-Bench_python_final.csv │ │ ├── Multi-SWE-Bench_python_final_perturbations.csv │ │ └── Multi-SWE-Bench_python_final_perturbations_qwen.csv │ │ │ ├── shap_analysis.ipynb │ ├── qwen_pert_java.ipynb │ └── qwen_pert_python_and_all_analysis.ipynb Annotation Data (RQ3) The annotation/ directory contains developer annotations used for RQ3. Each subfolder corresponds to one annotator. Files (*_annotations.json) include: Section-level importance Phrase-level importance Annotators are split across: Java bugs → annotator1, annotator2 Python bugs → annotator3, annotator4 Perturbation-Based Analysis (RQ1, RQ2, RQ3) All experimental data and analysis scripts are located in pert_based_analysis/. data/ Directory RQ1: Model Outputs (Patch Generation) patches/Generated patches for each bug instance across models. Java: Multi-SWE-Bench_java_Claude4.csv Multi-SWE-Bench_java_GPT_OSS.csv Multi-SWE-Bench_java_Qwen.csv Python: Multi-SWE-Bench_python_Claude4.csv Multi-SWE-Bench_python_GPT_OSS.csv Multi-SWE-Bench_python_Qwen.csv Base Bug Data Multi-SWE-Bench_java_final.csv — Java bug dataset Multi-SWE-Bench_python_final.csv — Python bug dataset RQ2: Perturbation Setup Multi-SWE-Bench_java_final_perturbations.csvGenerated perturbations for Java bugs Multi-SWE-Bench_python_final_perturbations.csvGenerated perturbations for Python bugs RQ2: Perturbation Results (Qwen) Multi-SWE-Bench_java_final_perturbations_qwen.csv Multi-SWE-Bench_python_final_perturbations_qwen.csv These contain model outputs after perturbing bug report components. RQ2: Attention Pattern Extraction Multi-SWE-Bench_java_qwen_attention_patterns.csv Multi-SWE-Bench_Python_Qwen_attention_patterns.csv These files contain computed attention patterns from perturbation analysis. Analysis Notebooks shap_analysis.ipynb→ RQ1 analysis (attention distribution and importance visualization) qwen_pert_java.ipynb→ RQ2 analysis for Java bugs qwen_pert_python_and_all_analysis.ipynb→ RQ2 + RQ3 analysis for Python bugs and combined results Mapping to Research Questions Research Question Description Files RQ1 Attention distribution across bug report sections patches/, model result CSVs, shap_analysis.ipynb RQ2 Attention patterns distinguishing success vs failure perturbation CSVs, attention pattern CSVs, Qwen notebooks RQ3 Alignment with developer attention annotation JSONs + combined analysis notebook Prompt Template We use a standardized prompt template for all experiments to ensure consistency across models and bugs. The prompt includes the buggy function and the corresponding bug report, and instructs the model to return only the corrected function. Python Prompt Template Given the following Python buggy function, fix the bug inside function based on the bug report provided. Return only the corrected function code, without extra explanation. ## Buggy Function <BUGGY_FUNCTION> ## Bug Report <BUG_REPORT> Java Prompt Template Given the following Java buggy function, fix the bug inside function based on the bug report provided. Return only the corrected function code, without extra explanation. ## Buggy Function <BUGGY_FUNCTION> ## Bug Report <BUG_REPORT>