The Artifacts for An Extensive Empirical Study of Nondeterministic Behavior in Static Analysis Tools.

This is the artifact for the ICSE 2025 paper, An Extensive Empirical Study of Nondeterministic Behavior in Static Analysis Tools. A pre-print of the original paper referencing this artifact can be found here: https://annabellam.github.io/docs/An_Extensive_Empirica_Study_of_Nondeterministic_Behavior_in_Static_A nalysis_Tools_-_Preprint.pdf. This artifact contains the code and data for the paper. The code contains the source code of the nondeterminism detection framework used in RQ2 as well as the instructions on how to run the framework to replicate the results reported in the paper. The data contains files that support the conclusions made in the two research questions (RQ1 and RQ2).

Changelog:

A Troubleshooting Section is added to address some potential issues, such as the docker permission issue Reviewer B encountered. We will continue updating this section whenever we notice any issues using our artifact.

A clarification is added to Section Data->RQ1 to avoid any confusion:

“There were 58 issues of Doop (as of the time this research was conducted) that are hosted on BitBucket, as stated at the beginning of Section III of our paper. These issues are not included in raw_data.zip as BitBucket requires administrative access to the repository to export the issues.”

The scripts to extract the GitHub issues and commits are included in our artifacts under the data/rq1/scripts folder.

A clarification is added to Section Detecting Nondeterminism Using Strategy I to avoid confusion when results_nondex/non_determinism folder is not generated:

"If no additional results are detected, the results_nondex/non_determinism folder will not be generated."