Replication package for the study "An Empirical Study of Java Code Improvements Based on Stack Overflow Answer Edits"

Matcha Replication Package This repository is the replication package for: An Empirical Study of Java Code Improvements Based on Stack Overflow Answer EditsIn-on Wiratsin, Chaiyong Ragkhitwetsagul, Matheus Paixao, Denis De Sousa, Pongpop Lapvikai, and Peter Haddawy. The study investigates how accepted Java answers on Stack Overflow change over time and whether edits to those answers can suggest useful improvements for Java code in open-source GitHub projects. The package contains the study setup, source-code analysis scripts, raw Siamese+ clone search results, manual validation results, the before–after static-analysis data and results, downloaded research corpora, and ethics approval documents. Study overview The study consists of four phases: Configure and extend Siamese. A ground-truth dataset of code clones between Stack Overflow and the Qualitas Corpus is used to tune the Siamese code-clone search tool. The extended tool, called Siamese+, supports searches across multiple Stack Overflow answer revisions and retrieval of the latest revision. Analyse Stack Overflow revisions and find candidate recommendations. Accepted Java answers with more than one revision are extracted from SOTorrent. Siamese+ searches for similarities between their revisions and Java methods in selected GitHub projects. The candidate pairs are then manually reviewed and classified. Measure the edits objectively. The manual classification of Phase 2 rests on human judgement, so this phase complements it with static analysis. Each original Stack Overflow snippet is compared against its latest revision using JavaParser (structural measures) and Checkstyle (readability), with paired Wilcoxon signed-rank tests and Holm correction. Apply recommendations in practice. Selected bug-fixing and code-improving recommendations are submitted to the corresponding GitHub projects as pull requests. This is a practical validation of the pipeline rather than a separately declared research question. The manuscript addresses the following research questions: RQ1: To what extent are accepted Java answers on Stack Overflow edited? RQ2: What types of code improvements do Stack Overflow answer edits suggest for code found in open-source projects, and how are they distributed across projects of different popularity? RQ3: What are the differences in terms of code structure and readability introduced by Stack Overflow answer edits? Phases 1 and 2 are supported by Study Setup/, RQ1 by RQ1/, RQ2 by RQ2/, and RQ3 (Phase 3) by RQ3/. Phase 4 outcomes are recorded in the pull-request columns of the manual validation results in RQ2/. Repository structure . ├── Ethics Approval/ │ ├── COE 2024-116.2203.pdf │ └── Participant Information Sheet.pdf ├── RQ1/ │ ├── .env │ └── SO_Snippet_Analysis/ │ ├── SOTorrentAnalyzer/ │ ├── analysis/ │ ├── files/ │ ├── StackOverflow_Snippets.zip │ ├── similarity.csv │ └── README.md ├── RQ2/ │ ├── Manual Validation Results/ │ ├── Manual_Classification_Codebook.docx │ ├── Manual_Classification_Codebook.pdf │ └── Siames+_results_2024-05-07_raw_result_FINAL.csv ├── RQ3/ │ └── before-after/ │ ├── dataset/ │ ├── results/ │ ├── tools/ │ ├── *.ipynb │ ├── build_*.py │ ├── exp_setup.md │ └── results_analysis*.md └── Study Setup/ ├── 1_GitHub/ └── 2_Siamese+/ Study Setup/1_GitHub Materials used to select and download the Java projects: File Description GH_Query_Results.csv Results returned by the GitHub project search process. GH_Selection.csv Final selection of 10,673 projects, including popularity-group variables. GH_Projects.csv Project metadata used during the collection process. GH_Projects.zip Archived source code of the collected GitHub projects. download_gh.py Script used to download projects and retain relevant Java files. log.csv Project-download log. Steps + Source Code for Matcha GH Project Selection.docx Original project-selection notes and source code. The project selection uses three popularity indicators: stargazers, forks, and watchers. The columns stars_region, forks_region, and watchers_region in GH_Selection.csv record the resulting popularity regions. GH_Projects.zip is approximately 8.6 GB compressed and expands to roughly 28 GB across 7.5 million files, because the archived projects retain their .git directories. Ensure sufficient disk space before extracting. Study Setup/2_Siamese+ Source code and configuration used to tune and run Siamese+: File or directory Description formatted_oracle.csv Filtered ground-truth clone pairs used for configuration optimisation. parameters/parameters_grid_search.yml Siamese parameter values used by the supplied configuration. config-index.properties Siamese indexing configuration template. config-search.properties Siamese search configuration template. siamese-0.0.6-SNAPSHOT.jar Packaged Siamese executable used by the scripts. elasticsearch-2.2.0.tar.gz Elasticsearch distribution used by Siamese. siamese_indexing.py Indexing orchestration. siamese_search.py Clone-search orchestration. generate_metrics.py Evaluation-metric calculation. requirements.txt Python dependencies for this part of the study. README.md Original execution notes. The optimised configuration reported in the manuscript is: Parameter Value n-gram size 16 Minimum clone size 7 Normalised QR percentile 3 Type-2 QR percentile 18 Type-1 QR percentile 10 Original-code QR percentile 1 Normalised-code boost 13 Type-2 boost 12 Type-1 boost 3 Original-code boost 17 Similarity thresholds 60%, 90%, 90%, 90% The Siamese+ scripts read filesystem locations from an .env file. The .env supplied with this package is located at RQ1/.env and contains relative dataset paths only; no credentials are included. Adjust its values for your environment before running the pipeline. RQ1/SO_Snippet_Analysis Materials supporting the analysis of Stack Overflow answer revisions: File or directory Description files/acceptedWithVersionAnswer.txt IDs of 140,840 accepted Java answers with multiple revisions. SOTorrentAnalyzer/ Java source for reading SOTorrent post-block histories and calculating revision statistics. StackOverflow_Snippets.zip Extracted Stack Overflow code snippets and revisions. similarity.csv Post-block similarity and revision-gap measurements. analysis/levenshtein_distances.csv Levenshtein distances between original and latest code-snippet revisions. analysis/code_lines_count.csv Java line counts for downloaded GitHub projects. analysis/projects_with_codelines.csv Project metadata combined with line counts and recommendation counts. analysis/improving_code.csv Improving-code recommendations and their manually assigned subtypes. analysis/analysis.py Project-group and recommendation analyses. analysis/code_diff.py Original-to-latest snippet edit-distance analysis. analysis/extract_answer_list.py Extraction of the accepted-answer list from a local SOTorrent database. analysis/java_rq1_stats.ipynb Notebook supporting RQ1 summary statistics. analysis/read_excel.ipynb Helper notebook for reading the manual validation spreadsheet. analysis/requirements.txt Python dependencies for the RQ1 analyses. README.md Detailed notes about the SOTorrent analysis pipeline. RQ2 Raw and manually reviewed Siamese+ recommendations: File Description Siames+_results_2024-05-07_raw_result_FINAL.csv Headerless raw Siamese+ output containing 793 GitHub-method/Stack-Overflow-snippet pairs. Manual Validation Results/Results_2024-05-07_manual_validation_FINAL.csv Manual validation and classification of the 793 candidate pairs. Manual Validation Results/Manual_Validation_Result_summary 25Aug25.xlsx Summary of the manual validation results. Manual_Classification_Codebook.docx Codebook defining the manual classification categories. Manual_Classification_Codebook.pdf PDF rendering of the codebook. Each row in the raw result contains two fields: The GitHub Java method path, method name, start line, and end line. The matched Stack Overflow snippet path, method name, start line, and end line. The manual-validation CSV records the associated GitHub project and Stack Overflow answer, validity checks, edit categories, applicability, recommendation type, implementation notes, source locations, cross-check results, pull-request status, and pull-request URL. The pull-request columns record the outcomes of Phase 4. RQ3/before-after Before–after static analysis of the Stack Overflow snippet edits (Phase 3). The 793 Siamese+ recommendations do not correspond to 793 distinct answers, because the same answer can be recommended to several projects and to several methods within one project. Deduplicating by the answer's code block reduces them to 209 unique snippet histories, of which 205 are eligible original–latest pairs. This deduplication avoids pseudoreplicating the same edit. Dataset File or directory Description dataset/before/, dataset/after/ Original and latest snapshots for the 205 eligible pairs. dataset/snippet_pairs.csv Provenance manifest for all unique snippet histories, with hashes, sizes, and eligibility status. dataset/study_pair_mapping.csv Maps all 793 study rows to the unique snippet histories. dataset/before_agress_yes/, dataset/after_agress_yes/ Snapshots for the 125-pair subset restricted to rows where validators agreed the recommendation was applicable. dataset/agress_yes_pairs.csv Manifest for the 125-pair subset. dataset/README.md Detailed dataset notes and counts. Snapshots are named so_<question-id>_block_<block-number>.java, which is the independent unit of the before–after analysis. Analysis and results File or directory Description tools/javaparser-core-3.28.1.jar JavaParser, used for the 19 structural measures. tools/checkstyle-13.9.0-all.jar Checkstyle, used for the readability measures. checkstyle-reduced.xml Checkstyle configuration defining the eight readability checks. build_javaparser_notebook_full.py, build_checkstyle_notebook_full.py Build the full-scope (205-pair) analysis notebooks. build_javaparser_notebook.py, build_checkstyle_notebook.py Build the 125-pair subset notebooks. build_javaparser_notebook_readable.py Build a readability-oriented JavaParser notebook variant. JavaParser_Before_After_Analysis_Full.ipynb, Checkstyle_Before_After_Analysis_Full.ipynb Full-scope analysis notebooks. JavaParser_Before_After_Analysis.ipynb, Checkstyle_Before_After_Analysis.ipynb Subset analysis notebooks. JavaParser_Before_After_Analysis_Readable.ipynb Readability-oriented JavaParser variant. prepare_dataset.rb Builds the before/after snapshot folders from the extracted revision files. exp_setup.md Design notes for the before–after evaluation. results_analysis.md, results_analysis_full.md Narrative summaries of the subset and full-scope results. requirements.txt Python dependencies for the RQ3 analyses. The results reported in the manuscript are those in the _full directories, which cover the full 205-pair scope: Directory Scope Pairs Analysed Parse-excluded results/javaparser_full/ Full (205 pairs) 205 128 77 results/checkstyle_full/ Full (205 pairs) 205 129 76 results/javaparser/ Agress Yes subset 125 80 45 results/checkstyle/ Agress Yes subset 125 81 44 results/javaparser_readable/ Agress Yes subset 125 80 45 The non-_full directories correspond to an earlier, narrower scope and are retained for transparency. Each results directory contains a *_run_summary.json recording the tool version, scope filter, pair counts, and configuration hashes, together with per-pair metrics, parse coverage, statistical summaries, and plots. A pair is measured only when its original and its latest revision share a common successful parse mode, which is why roughly 37% of pairs are parse-excluded: Stack Overflow snippets are frequently incomplete fragments. An exploratory PMD-based analysis was also attempted but is not part of the manuscript; see RQ3/before-after/exp_setup.md for why it was dropped. Ethics Approval Ethics approval and participant information for the pull-request phase of the study. Reproducing the analyses The package contains two kinds of reproducibility material: Reusable derived data, which can be analysed directly from the supplied CSV, XLSX, and JSON files. Collection and search pipelines, which require external datasets and older software infrastructure. 1. Analyse the supplied RQ1 data Requires Python 3.13 and the packages in RQ1/SO_Snippet_Analysis/analysis/requirements.txt: cd "RQ1/SO_Snippet_Analysis/analysis" python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt The principal RQ1 entry points are: RQ1/SO_Snippet_Analysis/analysis/java_rq1_stats.ipynb RQ1/SO_Snippet_Analysis/analysis/analysis.py RQ1/SO_Snippet_Analysis/analysis/code_diff.py Some original scripts retain absolute paths from the experiment environment. Change those paths to the corresponding locations in this package before execution. 2. Reproduce the RQ3 before–after analysis This stage is self-contained: the snapshots, tool JARs, and configurations are all included. It requires Python 3.13 and a JDK 17 or later: cd "RQ3/before-after" python3 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt Then execute the full-scope notebooks, which regenerate the contents of results/javaparser_full/ and results/checkstyle_full/: RQ3/before-after/JavaParser_Before_After_Analysis_Full.ipynb RQ3/before-after/Checkstyle_Before_After_Analysis_Full.ipynb The notebooks are generated artefacts. To rebuild them from source, run build_javaparser_notebook_full.py and build_checkstyle_notebook_full.py. The pandas, scipy, and matplotlib versions in RQ3/before-after/requirements.txt are pinned deliberately and must not be relaxed. scipy.stats.wilcoxon with method="auto" selects its p-value approximation based on the sample, and different scipy releases have been observed to choose differently for identical data and an identical test statistic. The published p-values correspond to the pinned versions. Rebuilding the dataset/before/ and dataset/after/ folders from the raw extracted revisions requires Ruby and prepare_dataset.rb; its default revision-root path must be changed, or supplied as the third command-line argument. This step is only necessary if you re-extract the Stack Overflow revisions in step 3, since the snapshots are already supplied. 3. Re-extract Stack Overflow revisions This stage requires: SOTorrent version 2020-12-31 loaded into MySQL; Java and a compatible MySQL JDBC driver; the SOTorrent tables queried by the Java source, particularly PostBlockVersion and PostBlockDiff; and RQ1/SO_Snippet_Analysis/files/acceptedWithVersionAnswer.txt. See RQ1/SO_Snippet_Analysis/README.md and RQ1/SO_Snippet_Analysis/SOTorrentAnalyzer/PostBlockProcessor.java for the original pipeline. Database credentials and filesystem paths must be configured for the local environment; the credentials embedded in the Java sources are placeholders for a local database and must be replaced. 4. Recreate the GitHub project corpus The original project list is in Study Setup/1_GitHub/GH_Selection.csv. The download script requires Python, GitPython, and access to GitHub: Study Setup/1_GitHub/download_gh.py Before running it, change its hard-coded output directory to a local destination. Repository metadata includes the last recorded commit SHA, which can be used to identify the collected revision where it remains available. 5. Run the Siamese+ setup The original Siamese+ pipeline was designed for Ubuntu and uses Java, Python, and Elasticsearch 2.2.0. Detailed original instructions and dependencies are provided in: Study Setup/2_Siamese+/README.md Study Setup/2_Siamese+/requirements.txt Configuration values that depend on local filesystem paths are loaded from an .env file. Do not reuse credentials or absolute paths from another environment. Known limitation: main.py and personalize_execution.py import a grid_search module that is not currently included in this package. Consequently, the full grid-search stage cannot be rerun from these entry points without restoring that module. The supplied optimised parameters and derived results remain available for inspection and downstream analysis. Main datasets and record counts Dataset Records Accepted Java answers with multiple revisions 140,840 Original/latest snippet pairs with Levenshtein distances 233,825 Selected GitHub projects 10,673 Raw Siamese+ candidate pairs 793 Manually validated candidate pairs 793 Unique Stack Overflow snippet histories (RQ3) 209 Eligible original–latest pairs (RQ3) 205 Pairs analysed by JavaParser (RQ3) 128 Pairs analysed by Checkstyle (RQ3) 129 The raw candidate pairs contain 397 unique GitHub repositories and 208 unique Stack Overflow answers. All 208 Stack Overflow answer IDs represented in the raw output also occur in the manual-validation dataset. Software and external data The complete pipeline depends on external or third-party resources: SOTorrent 2020-12-31; Stack Overflow content and revision history; the Qualitas Corpus release 20130901r; GitHub repositories selected for the study; Siamese; Elasticsearch 2.2.0; a MySQL JDBC driver; JavaParser 3.28.1 (Apache-2.0 / LGPL-3.0 dual licence); and Checkstyle 13.9.0 (LGPL-2.1). Users are responsible for complying with the licenses and terms associated with these resources. The GitHub project archive contains projects with their own licenses; consult the license information within each project before reuse. Known portability notes Some scripts preserve absolute paths from the original experiment machines and must be configured before reuse. This affects the RQ1 analysis scripts, the Siamese+ scripts, and RQ3/before-after/prepare_dataset.rb. The raw Siamese+ result is intentionally retained in its original, headerless format. analysis/levenshtein_distances.csv contains absolute source paths from the original machine. The Stack Overflow answer and block identifiers embedded in the paths are the portable identifiers. RQ3/before-after/dataset/snippet_pairs.csv similarly records absolute Original Source Path and After Source Path values from the extraction machine. The package-relative Before Dataset Path and After Dataset Path columns in the same file are the portable references, and the analysis notebooks use those. Four selected projects do not have entries in analysis/code_lines_count.csv; users reproducing the line-count analysis should consult the download log and verify the availability and contents of those projects. The study uses historical Stack Overflow and GitHub snapshots. Results should not be interpreted as describing the current state of either platform. Data availability and citation This replication package is published on Zenodo. Cite the concept DOI, which always resolves to the most recent version: https://doi.org/10.5281/zenodo.17220745 Version-specific DOIs are listed on the Zenodo record and should be used only when referring to one particular version of the package. When citing the package, use the citation metadata displayed on the published Zenodo record. When citing the research findings, cite the associated manuscript. Contact For questions about the replication package, contact: Chaiyong RagkhitwetsagulFaculty of Information and Communication Technology, Mahidol UniversityEmail: [e-mail]