ARcode: HPC Application Recognition Through Image-encoded Monitoring Data

This tar file contains the docker image for building the ARcode model and baseline models for application recognition for the SC22 paper with the same title. The files/folders in this image contains: notebooks: The notebooks for models and experiment results.-- ARcode.ipynb: The interactive Jupyter Notebook for the ARcode model.-- ARcode_unknown.ipynb: The interactive Jupyter Notebook for the ARcode model for detecting unknown applications.-- ARcode_partial.ipynb: The interactive Jupyter Notebook for the ARcode model on partial job signatures.-- ARcode_channel.ipynb: The interactive Jupyter Notebook for the ARcode model on one channel of job signatures.-- baselines.ipynb: The interactive Jupyter Notebook for the baseline models. These models are Random Forest, LinearSVC and SVC; all of them are implemented through Taxonomist(https://doi.org/10.6084/m9.figshare.6384248.v1).-- baselines_unknown.ipynb: The interactive Jupyter Notebook for the baseline models for detecting unknown applications. dataset: The dataset for training the models mentioned above.-- ARcode_labels.npy: A numpy array of the signatures' labels.-- ARcode_signatures.npy: A numpy array of the generated signatures.-- baseline_labels.npy: A numpy array of the labels for the baseline dataset.-- baseline_features.npy: A numpy array of the statistic features generated from the raw monitoring data.-- knl_app_code.json: Mapping of IDs to application names. This mapping is used when creating the dataset. models: The saved models.-- arcode.h5: An HDF5 file containing the serialized weights for the ARcode model.-- arcode.json: A JSON file describing the ARcode model. results: The saved experiment results. Following these steps to start Jupyter Notebook in the image: 1. Load the image into Docker on your local machine:docker load < archive-arcode.tar 2. Start the Jupyter notebook in the docker image:docker run --init --user root -p 8888:8888 artlands/arcode 3. Copy the URL shown in your terminal and paste in a brower: http://127.0.0.1:8888/?token=your_token Acknowledgement: This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231

本tar归档文件包含用于构建ARcode模型与基线模型的Docker镜像，用于复现本篇同名2022年国际超级计算大会（SC22）论文中的应用识别相关研究。镜像内的文件与文件夹说明如下：
### notebooks文件夹
存放模型代码与实验结果的交互式脚本文件夹：
- ARcode.ipynb：ARcode模型的交互式Jupyter Notebook脚本；
- ARcode_unknown.ipynb：用于检测未知应用的ARcode模型交互式Jupyter Notebook脚本；
- ARcode_partial.ipynb：针对部分作业特征签名（job signatures）的ARcode模型交互式Jupyter Notebook脚本；
- ARcode_channel.ipynb：针对单通道作业特征签名的ARcode模型交互式Jupyter Notebook脚本；
- baselines.ipynb：基线模型的交互式Jupyter Notebook脚本，所涉基线模型包括随机森林（Random Forest）、LinearSVC与SVC，所有模型均通过Taxonomist（https://doi.org/10.6084/m9.figshare.6384248.v1）实现；
- baselines_unknown.ipynb：用于检测未知应用的基线模型交互式Jupyter Notebook脚本。
### dataset文件夹
用于训练上述模型的数据集文件夹：
- ARcode_labels.npy：存储作业特征标签的NumPy数组文件；
- ARcode_signatures.npy：存储生成的作业特征签名的NumPy数组文件；
- baseline_labels.npy：存储基线数据集标签的NumPy数组文件；
- baseline_features.npy：存储从原始监控数据中提取的统计特征的NumPy数组文件；
- knl_app_code.json：应用ID与名称的映射文件，用于数据集构建流程中。
### models文件夹
预训练好的模型文件：
- arcode.h5：存储ARcode模型序列化权重的HDF5格式文件；
- arcode.json：描述ARcode模型结构的JSON格式文件。
### results文件夹
存储实验结果的文件夹。
### 镜像启动Jupyter Notebook步骤
1. 在本地Docker环境加载该归档镜像：`docker load < archive-arcode.tar`
2. 在Docker容器中启动Jupyter Notebook服务：`docker run --init --user root -p 8888:8888 artlands/arcode`
3. 复制终端中显示的访问URL并粘贴至浏览器即可打开交互界面：`http://127.0.0.1:8888/?token=your_token`
### 致谢
本研究使用了美国能源部科学用户设施国家能源研究科学计算中心（NERSC, National Energy Research Scientific Computing Center）的计算资源，该中心隶属于劳伦斯伯克利国家实验室，运行合同编号为DE-AC02-05CH11231。