Temporal validity of software datasets for code metrics: an empirical assessment of sampling strategies

This is the repository for the scripts and data of the study "Building and updating software datasets: an empirical assessment". Data collected The data generated for the study it can be downloaded as a zip file. Each folder inside the file corresponds to one of the datasets of projects employed in the study (qualitas, currentSample and qualitasUpdated). Every dataset comprised three files "class.csv", "method.csv" and "sample.csv", with class metrics, method metrics and repository metadata of the projects respectively. Here is a description of the datasets: qualitas: includes code metrics and repository metrics from the projects in the release 20130901r of the Qualitas Corpus. currentSample: includes code metrics and repository metrics from a recent sample collected with our sampling procedure. qualitasUpdated: includes code metrics and repository metrics from an updated version of the Qualitas Corpus applying our maintenance procedure. Plot graphics To plot the results and graphics in the article there is a Jupyter Notebook "Experiment.ipynb". It is initially configured to use the data in "datasets" folder. Replication Kit For replication purposes, the datasets containing recent projects from Github can be re-generated. To do so, the virtual environment must have installed the dependencies in "requirements.txt" file, add Github's tokens in "./token" file, re-define or leave as is the paths declared in the constants (variables written in caps) in the main method, and finally run "main.py" script. The portable versions of the source code scanner Sourcemeter are located as zip files in "./Sourcemeter/tool" directory. To install Sourcemeter the appropriate zip file must be decompressed excluding the root folder "SourceMeter-10.2.0-x64-<OS>". The script comprise 5 steps: Project retrieval from Github: at first the sampling frame with projects complying with a specific quality criteria are retrieved from Github's API. Create samples: with the sampling frame retrieved, the current samples are selected (currentSample and qualitasUpdated). In the case of qualitasUpdated, it is important to have first the "sample.csv" file inside the qualitas folder of the dataset originally created for the study. This file contains the metadata of the projects in Qualitas Corpus. Project download and analysis: when all the samples are selected from the sampling frame (currentSample and qualitasUpdated), the repositories are downloaded and scanned with SourceMeter. In the cases in which the analysis is not possible, the projects are replaced with another one with similar size. Outlier detection: once the datasets are collected, it is necessary to manually look for possible outliers in the code metrics under study. In the notebook "Experiment.ipynb" there are specific sections dedicated for it ("Outlier detection (Section 4.2.2)"). Outlier replacement: when the outliers are detected, in the same notebook there is also a section for outlier replacement ("Replace Outliers") where the outliers' url have to be listed to find the appropriate replacement. If it is required, the metrics from the Qualitas Corpus can also be re-generated. First, it is necessary to download the release 20130901r from its official webpage. Second, decompress the .tar files downloaded. Third, make sure that the compressed files with source code from the projects (.java files) are placed in the "compressed" folder, in some cases it is necessary to read the "QC_README" file in the project's folder. Finally, run the original main script "Generate metrics for the Qualitas Corpus (QC) dataset" part of the code.本仓库用于承载研究《构建与更新软件数据集：一项实证评估》的配套脚本与数据集。 ### 已采集数据集 本研究生成的数据集可通过压缩包下载。压缩包内的每个文件夹对应本研究使用的三类项目数据集：qualitas、currentSample与qualitasUpdated。每类数据集均包含三个文件：class.csv、method.csv与sample.csv，分别存储项目的类度量指标、方法度量指标与仓库元数据。各数据集详情如下： - qualitas：采用Qualitas语料库（Qualitas Corpus）20130901r版本中的项目代码度量指标与仓库度量指标。 - currentSample：采用通过本研究采样流程采集的近期项目样本的代码度量指标与仓库度量指标。 - qualitasUpdated：采用经过本研究维护流程更新后的Qualitas语料库版本中的项目代码度量指标与仓库度量指标。 ### 图表生成 若需生成论文中的结果与图表，可使用Jupyter笔记本（Jupyter Notebook）文件"Experiment.ipynb"。该文件初始配置为读取datasets文件夹内的数据集。 ### 复现套件 为实现研究复现，可重新生成包含GitHub近期项目的数据集。具体操作步骤如下：首先在虚拟环境中安装requirements.txt文件内声明的依赖项；其次将GitHub访问令牌添加至./token文件中；随后可按需修改或保留main方法内常量（大写命名的变量）所定义的路径；最后运行main.py脚本。 源代码扫描工具Sourcemeter的便携版以压缩包形式存储于./Sourcemeter/tool目录下。安装Sourcemeter时，需解压对应压缩包，且不得保留根文件夹SourceMeter-10.2.0-x64-<OS>。 该脚本包含以下5个执行步骤： 1. 项目检索（从GitHub）：首先通过GitHub API检索符合特定质量标准的项目采样框架。 2. 样本构建：基于已检索到的采样框架，选取当前样本集（currentSample与qualitasUpdated）。其中针对qualitasUpdated数据集，需优先获取本研究初始生成的qualitas数据集中的sample.csv文件，该文件存储了Qualitas语料库中项目的元数据。 3. 项目下载与分析：完成所有样本集（currentSample与qualitasUpdated）的选取后，下载对应仓库并使用Sourcemeter进行扫描。若部分项目无法完成分析，则使用规模相近的其他项目替代。 4. 异常值检测：数据集采集完成后，需手动排查所研究的代码度量指标中可能存在的异常值。"Experiment.ipynb"文件中设有专门的章节“异常值检测（4.2.2节）”用于完成该操作。 5. 异常值替换：检测到异常值后，可在该Notebook的“替换异常值”章节中完成异常值替换操作，需将异常值对应的仓库URL填入对应位置以检索合适的替代项目。 若需重新生成Qualitas语料库的度量指标，可按以下步骤操作： 1. 从其官方页面下载20130901r版本的压缩包； 2. 解压下载的.tar文件； 3. 确保项目源代码压缩文件（.java格式）存放至compressed文件夹内，部分场景下需读取项目文件夹中的QC_README文件； 4. 最后运行原主脚本“为Qualitas语料库（QC）数据集生成度量指标”模块。