Static analysis evaluation experiment data

This repository contains the experiment artifacts for our paper entitled “Comprehensive Evaluation of Static Analysis Tools for Their Performance in Finding Vulnerabilities in Java Code” submitted to the IEEE Access Journal.

For each part of our experiment, we mention the related file name in this repository.

Research Methodology:

A. Experiment design: no related documents.

B. Preparing the Juliet Test Suite: 

In this section, we prepared Juliet for being analyzed by the five tools of the study. The related document for this section is called juliet_preparation.pdf.

C. Evaluation metrics: no related documents.

D. Experiment execution: 

Step 1: For each of the five tools, review the documentation to identify and activate the related checker(s).

The related documents of this section are pmd_checkers.pdf. spotbugs_fsb_checkers.pdf, infer_checkers.pdf, and sonar_checkers.pdf. Those documents include all the checkers that have been used and activated (if they were not active by default) to enable the Juliet analysis using the relevant checkers.

Step 2: Run each tool on each CWE and get the output reports.

The related document is called running_the_tools.pdf, which includes the detailed steps for running each tool.

Step 3: For each tool, and each CWE, consider the relevant checker's results.

no related document.

Step 4: For each tool, and each CWE, compute TP, FP, TN, and FN.

The related document is response_variables.xls

Step 5: Compute the response variables for each tool detecting each CWE.

The related document is response_variables.xls

Step 6: For each tool, compute collective evaluation metrics.

The related document is response_variables.xls

本仓库包含我们提交至IEEE Access期刊的论文《针对静态分析工具在Java代码漏洞检测中性能表现的综合评估》的实验制品。

针对本实验的各个环节，我们均在本仓库中标注了对应的相关文件名。

研究方法：

A. 实验设计：无相关文档。

B. 准备Juliet测试套件（Juliet Test Suite）：
本环节中，我们完成了Juliet测试套件的适配配置，使其可被本研究涉及的五款静态分析工具进行漏洞检测分析，本环节的相关文档为`juliet_preparation.pdf`。

C. 评估指标：无相关文档。

D. 实验执行：
步骤1：针对五款工具中的每一款，查阅其官方文档以确定并启用对应的检查器（checker）。本环节的相关文档包括`pmd_checkers.pdf`、`spotbugs_fsb_checkers.pdf`、`infer_checkers.pdf`以及`sonar_checkers.pdf`，上述文档收录了本次研究中使用并启用的全部检查器（若默认未启用则需手动激活），以确保可通过对应检查器完成Juliet测试套件的分析工作。
步骤2：针对每一项通用弱点枚举（CWE），运行每一款工具并生成输出报告。本环节的相关文档为`running_the_tools.pdf`，其中收录了运行每一款工具的详细操作步骤。
步骤3：针对每一款工具及每一项CWE，梳理对应检查器的检测结果。无相关文档。
步骤4：针对每一款工具及每一项CWE，计算真阳性（TP）、假阳性（FP）、真阴性（TN）及假阴性（FN）。本环节的相关文档为`response_variables.xls`。
步骤5：针对每一款工具检测的每一项CWE，计算其响应变量。本环节的相关文档为`response_variables.xls`。
步骤6：针对每一款工具，计算其整体评估指标。本环节的相关文档为`response_variables.xls`