Replication Package: Unboxing Default Argument Breaking Changes in Scikit Learn

<strong>Replication Package</strong> This repository contains data and source files needed to replicate our work described in the paper "Unboxing Default Argument Breaking Changes in Scikit Learn". <strong>Requirements</strong> We recommend the following requirements to replicate our study: Internet access At least 100GB of space Docker installed Git installed <strong>Package Structure</strong> We relied on Docker containers to provide a working environment that is easier to replicate. Specifically, we configure the following containers: <code>data-analysis</code>, an R-based Container we used to run our data analysis. <code>data-collection</code>, a Python Container we used to collect Scikit's default arguments and detect them in client applications. <code>database</code>, a Postgres Container we used to store clients' data, obtainer from Grotov et al. <code>storage</code>, a directory used to store the data processed in <code>data-analysis</code> and <code>data-collection</code>. This directory is shared in both containers. <code>docker-compose.yml</code>, the Docker file that configures all containers used in the package. In the remainder of this document, we describe how to set up each container properly. <strong>Using VSCode to Setup the Package</strong> We selected VSCode as the IDE of choice because its extensions allow us to implement our scripts directly inside the containers. In this package, we provide configuration parameters for both <code>data-analysis</code> and <code>data-collection</code> containers. This way you can directly access and run each container inside it without any specific configuration. You first need to set up the containers <pre><code>$ cd /replication/package/folder $ docker-compose build $ docker-compose up # Wait docker creating and running all containers </code></pre> Then, you can open them in Visual Studio Code: Open VSCode in project root folder Access the command palette and select "Dev Container: Reopen in Container" Select either <em>Data Collection</em> or <em>Data Analysis</em>. Start working If you want/need a more customized organization, the remainder of this file describes it in detail. <strong>Longest Road: Manual Package Setup</strong> <strong>Database Setup</strong> The database container will automatically restore the dump in <code>dump_matroskin.tar</code> in its first launch. To set up and run the container, you should: Build an image: <pre><code>$ cd ./database $ docker build --tag 'dabc-database' . $ docker image ls REPOSITORY TAG IMAGE ID CREATED SIZE dabc-database latest b6f8af99c90d 50 minutes ago 18.5GB </code></pre> Create and enter inside the container: <pre><code class="language-bash">$ docker run -it --name dabc-database-1 dabc-database $ docker exec -it dabc-database-1 /bin/bash root# psql -U postgres -h localhost -d jupyter-notebooks jupyter-notebooks=# \dt List of relations Schema | Name | Type | Owner --------+-------------------+-------+------- public | Cell | table | root public | Code_cell | table | root public | Md_cell | table | root public | Notebook | table | root public | Notebook_features | table | root public | Notebook_metadata | table | root public | repository | table | root </code></pre> If you got the tables list as above, your database is properly setup. It is important to mention that this database is extended from the one provided by Grotov et al.. Basically, we added three columns in the table <code>Notebook_features</code> (<code>API_functions_calls</code>, <code>defined_functions_calls</code>, and<code>other_functions_calls</code>) containing the function calls performed by each client in the database. <strong>Data Collection Setup</strong> This container is responsible for collecting the data to answer our research questions. It has the following structure: <code>dabcs.py</code>, extract DABCs from Scikit Learn source code, and export them to a CSV file. <code>dabcs-clients.py</code>, extract function calls from clients and export them to a CSV file. We rely on a modified version of Matroskin to leverage the function calls. You can find the tool's source code in the `matroskin`` directory. <code>Makefile</code>, commands to set up and run both <code>dabcs.py</code> and <code>dabcs-clients.py</code> <code>matroskin</code>, the directory containing the modified version of matroskin tool. We extended the library to collect the function calls performed on the client notebooks of Grotov's dataset. <code>storage</code>, a docker volume where the data-collection should save the exported data. This data will be used later in Data Analysis. <code>requirements.txt</code>, Python dependencies adopted in this module. Note that the container will automatically configure this module for you, e.g., install dependencies, configure matroskin, download scikit learn source code, etc. For this, you must run the following commands: <pre><code class="language-bash">$ cd ./data-collection $ docker build --tag "data-collection" . $ docker run -it -d --name data-collection-1 -v $(pwd)/:/data-collection -v $(pwd)/../storage/:/data-collection/storage/ data-collection $ docker exec -it data-collection-1 /bin/bash $ ls Dockerfile Makefile config.yml dabcs-clients.py dabcs.py matroskin storage requirements.txt utils.py </code></pre> If you see project files, it means the container is configured accordingly. <strong>Data Analysis Setup</strong> We use this container to conduct the analysis over the data produced by the Data Collection container. It has the following structure: <code>dependencies.R</code>, an R script containing the dependencies used in our data analysis. <code>data-analysis.Rmd</code>, the R notebook we used to perform our data analysis <code>datasets</code>, a docker volume pointing to the <code>storage</code> directory. Execute the following commands to run this container: <pre><code class="language-bash">$ cd ./data-analysis $ docker build --tag "data-analysis" . $ docker run -it -d --name data-analysis-1 -v $(pwd)/:/data-analysis -v $(pwd)/../storage/:/data-collection/datasets/ data-analysis $ docker exec -it data-analysis-1 /bin/bash $ ls data-analysis.Rmd datasets dependencies.R Dockerfile figures Makefile </code></pre> If you see project files, it means the container is configured accordingly. A note on <code>storage</code> shared folder As mentioned, the <code>storage</code> folder is mounted as a volume and shared between <code>data-collection</code> and <code>data-analysis</code> containers. We compressed the content of this folder due to space constraints. Therefore, before starting working on Data Collection or Data Analysis, make sure you extracted the compressed files. You can do this by running the <code>Makefile</code> inside <code>storage</code> folder. <pre><code class="language-bash">$ make unzip # extract files $ ls clients-dabcs.csv clients-validation.csv dabcs.csv Makefile scikit-learn-versions.csv versions.csv $ make zip # compress files $ ls csv-files.tar.gz Makefile</code></pre>

<strong>复现包（Replication Package）</strong> 本仓库包含复现论文《解析Scikit Learn中的默认参数破坏性变更》（Unboxing Default Argument Breaking Changes in Scikit Learn）中所述研究所需的数据与源代码文件。
<strong>环境要求</strong> 为复现本研究，我们推荐以下配置：可访问互联网、至少100GB可用存储空间、已安装Docker、已安装Git。
<strong>套件结构</strong> 本套件依托Docker容器构建可便捷复现的运行环境，具体配置了以下容器：
<code>data-analysis</code>：基于R语言的容器，用于执行数据分析任务。
<code>data-collection</code>：基于Python的容器，用于采集Scikit Learn的默认参数并在客户端应用中检测此类参数。
<code>database</code>：基于PostgreSQL的容器，用于存储从Grotov等人的研究中获取的客户端数据。
<code>storage</code>：存储目录，用于存放<code>data-analysis</code>与<code>data-collection</code>处理后的数据，该目录在两个容器中共享挂载。
<code>docker-compose.yml</code>：用于配置本套件中所有容器的Docker配置文件。
本文后续部分将详细说明各容器的正确配置方式。
<strong>使用VSCode配置套件</strong> 我们选用VSCode作为集成开发环境（IDE），因其扩展插件支持直接在容器内部编写脚本。本套件已为<code>data-analysis</code>与<code>data-collection</code>容器预置配置参数，用户可直接在VSCode中访问并运行对应容器，无需额外配置。
首先完成容器配置：
<pre><code>$ cd /replication/package/folder
$ docker-compose build
$ docker-compose up # 等待Docker创建并启动所有容器
</code></pre>
随后可在Visual Studio Code中打开容器：
1. 在项目根目录打开VSCode；
2. 调出命令面板并选择「开发容器：在容器中重新打开」；
3. 选择「数据采集（Data Collection）」或「数据分析（Data Analysis）」；
4. 开始工作。
若需自定义配置，本文后续部分将提供详细说明。
<strong>手动配置套件：完整流程</strong>
<strong>数据库容器配置</strong> 数据库容器首次启动时将自动恢复<code>dump_matroskin.tar</code>中的数据备份。如需手动配置并运行容器，请执行以下步骤：
构建镜像：
<pre><code class="language-bash">$ cd ./database
$ docker build --tag 'dabc-database' .
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
dabc-database latest b6f8af99c90d 50 minutes ago 18.5GB
</code></pre>
创建并进入容器：
<pre><code class="language-bash">$ docker run -it --name dabc-database-1 dabc-database
$ docker exec -it dabc-database-1 /bin/bash
root# psql -U postgres -h localhost -d jupyter-notebooks
jupyter-notebooks=# dt
List of relations
Schema | Name | Type | Owner
--------+-------------------------+-------+-------
public | Cell | table | root
public | Code_cell | table | root
public | Md_cell | table | root
public | Notebook | table | root
public | Notebook_features | table | root
public | Notebook_metadata | table | root
public | repository | table | root
</code></pre>
若能获取如上所示的表列表，则说明数据库配置成功。需说明的是，本数据库基于Grotov等人提供的数据库扩展而来：我们在<code>Notebook_features</code>表中新增了三列，分别为<code>API_functions_calls</code>、<code>defined_functions_calls</code>与<code>other_functions_calls</code>，用于存储数据库中每个客户端执行的函数调用记录。
<strong>数据采集容器配置</strong> 该容器负责采集数据以解答本研究的相关问题，其内部结构如下：
<code>dabcs.py</code>：从Scikit Learn源代码中提取默认参数破坏性变更（DABC），并将结果导出为CSV文件。
<code>dabcs-clients.py</code>：从客户端代码中提取函数调用，并将结果导出为CSV文件。
我们使用了修改版的Matroskin工具以提取函数调用，该工具的源代码存放于<code>matroskin</code>目录中。
<code>Makefile</code>：用于配置并运行<code>dabcs.py</code>与<code>dabcs-clients.py</code>的命令脚本。
<code>matroskin</code>目录：存放修改版Matroskin工具的源代码，我们对该库进行了扩展，以采集Grotov数据集的客户端笔记本中的函数调用记录。
<code>storage</code>：Docker卷目录，用于存放数据采集容器导出的数据，该数据将用于后续的数据分析流程。
<code>requirements.txt</code>：本模块所需的Python依赖列表。
请注意，容器将自动完成本模块的配置，包括安装依赖、配置Matroskin工具、下载Scikit Learn源代码等。需执行以下命令完成配置：
<pre><code class="language-bash">$ cd ./data-collection
$ docker build --tag "data-collection" .
$ docker run -it -d --name data-collection-1 -v $(pwd)/:/data-collection -v $(pwd)/../storage/:/data-collection/storage/ data-collection
$ docker exec -it data-collection-1 /bin/bash
$ ls
Dockerfile Makefile config.yml dabcs-clients.py dabcs.py matroskin storage requirements.txt utils.py
</code></pre>
若能看到如上所示的项目文件，则说明容器配置成功。
<strong>数据分析容器配置</strong> 本容器用于对数据采集容器生成的数据执行分析任务，其内部结构如下：
<code>dependencies.R</code>：包含本数据分析任务所需依赖的R脚本。
<code>data-analysis.Rmd</code>：用于执行数据分析的R笔记本文件。
<code>datasets</code>：指向<code>storage</code>目录的Docker卷目录。
需执行以下命令以运行该容器：
<pre><code class="language-bash">$ cd ./data-analysis
$ docker build --tag "data-analysis" .
$ docker run -it -d --name data-analysis-1 -v $(pwd)/:/data-analysis -v $(pwd)/../storage/:/data-collection/datasets/ data-analysis
$ docker exec -it data-analysis-1 /bin/bash
$ ls
data-analysis.Rmd datasets dependencies.R Dockerfile figures Makefile
</code></pre>
若能看到如上所示的项目文件，则说明容器配置成功。
<strong>关于共享存储目录<code>storage</code>的说明</strong> 如前所述，<code>storage</code>目录以卷的形式挂载，并在<code>data-collection</code>与<code>data-analysis</code>容器之间共享。受存储空间限制，我们对该目录的内容进行了压缩。因此，在开始数据采集或数据分析任务前，请务必先解压压缩文件。你可通过在<code>storage</code>目录中执行<code>Makefile</code>来完成解压：
<pre><code class="language-bash">$ make unzip # extract files
$ ls
clients-dabcs.csv clients-validation.csv dabcs.csv Makefile scikit-learn-versions.csv versions.csv
$ make zip # compress files
$ ls
csv-files.tar.gz Makefile
</code></pre>