Supporting Error Chains in Static Analysis for Precise Evaluation Results and Enhanced Usability

Abstract<b>Context:</b> Static analyses are well-established to aid in understanding bugs or vulnerabilities during the development process or in large-scale studies. A low false positive rate is essential for the adaption in practice and for precise results of empirical studies. Unfortunately, static analyses tend to report where a vulnerability manifests rather than the fix location. This can cause presumed false positives or imprecise results.<b> </b><b>Method:</b> To address this problem, we designed an adaption of an existing static analysis algorithm that can distinguish between a manifestation and fix location and reports error chains. Each error chain presents the dependency between the fix location with at least one manifestation location. We used our tool for a case study of 471 GitHub repositories and conducted an expert interview to investigate usability implications of the change. Further, we benchmarked both analysis versions to compare the runtime impact.<b> Result:</b> We found that 50% of the projects with a report had at least one error chain. During our expert interview, all participants required fewer executions of the static analysis if they used our adapted version. Our performance benchmark demonstrated that our improvement caused only a minimal runtime overhead of less than 4%.<b> </b><b>Conclusion:</b> Our results indicate that error chains occur frequently in real-world projects and ignoring them can lead to imprecise evaluation results. The performance benchmark indicates that our tool is a feasible and efficient solution for detecting error-chains in real-world projects. Further, our results indicate that the usability of static analyses benefits from supporting error chains.DataThis artefact contains additional information for our evaluation.<br>Folder <code>code_study</code> (RQ1)The folder <code>JavaCryptographicAchitecture_BET</code> contains the CrySL rules for the JCA that we used for the code study.The file <code>SUBS.jar</code> is the version of SUBS that we used for our code study.The file <code>README.md</code> describes how to use the Docker image for scanning the code with CogniCryptSUBS.The file <code>CREDENTIALS.txt</code> is a dummy file for the GitHub tokens required for the analysis.The file <code>run_cc_subs.sh</code> is a helper script to execute CogniCryptSUBS and used by the Docker container.The file `Dockerfile` is the Docker image used for the code study.Folder <code>performance_analysis</code> (RQ2)The folder <code>1_run_performance_analysis/JavaCryptographicArchitecture</code> contains the CrySL rules for the JCA that we used for the benchmark that do not support Backward Error Tracking (BET).The folder <code>1_run_performance_analysis/JavaCryptographicArchitecture_BET</code> contains the CrySL rules for the JCA that we used for the benchmark that support BET.The different <code>1_run_performance_analysis/*.jar</code> files are the different evaluated versions of CogniCrypt and CogniCrypt_SUBS.The file <code>1_run_performance_analysis/Dockerfile</code> is the Docker image used to execute the benchmark.The file <code>1_run_performance_analysis/run_performance_analysis.sh</code> includes the commands to execute the different tools on our benchmark and the different target folders for the different configurations/groups of the benchmark.The folder <code>2_parse_results/data</code> contains the results obtained for the five different configurations for the different tools.The file <code>2_parse_results/generate_graphics.py</code> generates the graphics used in the paper.The folder <code>results</code> contains the graphics, such as Fig. 4, for the different configurations.Folder <code>expert_interview</code> (RQ3)The code examples for task 1 and 2 are in the folder <code>expertinterview_examplecode1</code> and <code>expert interview_examplecode2</code>, respectively.The invitation and questions are in the file `expert interview.md`.An overview of the obtained results are in the file `expert interview_results.csv`.Further, we include the graphics for the runtime evaluation as pdf-files.ChangesVersion 2: Restructure the main folder to include one folder for each research question answered in the paper. Further, added data for the code study and more details for the performance benchmark.Version 1: Add details for the expert interview and pdf-files for the performance benchmark. All files were added to the main folder.