Enhanced Westermo dataset - Transformed and Modified for Test case Selection and Priorotization in the context of Continuous Integration and Reinforcement Learning.

<strong>Overview</strong> This repository contains a modified version of the existing, recently published dataset, Westermo. The initial dataset was gathered at Westermo Network Technologies AB, located in Västerås, Sweden. It encompasses over <strong>1 Million verdicts</strong> obtained from testing embedded systems, collected over a span of more than <strong>500 consecutive days</strong> of nightly testing. The dataset has been transformed and tailored specifically to cater to the research community, particularly for addressing challenges such as regression test selection, identification of flaky tests, and visualization of test results. The original dataset can be accessed through the reference provided in <strong>[1]</strong>. The Westermo dataset offers valuable historical information regarding the execution of test cases and their corresponding results. It serves as a valuable resource for evaluating and comparing different Test case Selection and Prioritization (TSP) techniques, enabling researchers to identify test cases that are more likely to fail during subsequent executions. Test cases in the dataset are characterized by attributes such as execution duration, previous last execution time, and the results of their recent executions. This dataset offers valuable historical information regarding the execution of test cases and their corresponding results. It serves as a valuable resource for evaluating and comparing different test case prioritization and selection techniques, enabling researchers to identify test cases that are more likely to fail during subsequent executions. Test cases in the dataset are characterized by attributes such as execution duration, previous last execution time, and the results of their recent executions. <strong>Table 1: Dataset Overview</strong> Test Cases 1855 CI Cycles 15,197 Verdict 1,036,818 Failed 5.03% However, the diversity and multitude of the features in the dataset can be irrelevant to some TSP approaches. This led us to perform a dataset conversion, where we customized Westermo to have the same features from Paint Control and IOF/ROL, two widely used datasets in Reinforcement Learning based TSP approaches. This conversion required the combination of multiple variables and generating the target ones. When it comes to generating the “LastResults” and “Cycle” values, further analysis was required and the data handling needed an in-depth understanding of how the nightly testing was conducted. This led us to investigate what a CI cycle is in their context, and we followed their definition of a session, stating that “a session is when we run a suite of tests on one test system with a certain software version and testware version”. When splitting the data according to the 9 different systems used, we were able to generate 9 different sub-sets that fit the CI context. <strong>File Format</strong> The compressed .zip file contains 9 files, each one corresponding to each of the 9 systems. The datasets are available in CSV format, with the semicolon (;) serving as the delimiter. The columns included are represented in the table below along with their descriptions. <strong>Table 2: Parameters of the dataset</strong> Column Name Content jid job id, together with the system name, the pair (jid, system) forms a unique key for a test session System Name of the test system Name Unique numeric identifier of the test case Verdict Test verdict of this test execution (Failed: 1, Passed: 0) Duration Approximated runtime of the test case Cycle The number of the CI cycle this test execution belongs to. Group The group test case belongs to. LastRun Previous last execution of the test case as date-time-string (Format: <em>YYYY-MM-DD HH:ii </em>) Id Unique numeric identifier of the test execution CalcPrio Priority of the test case, calculated by the prioritization algorithm (output column, initially 0) result_array List of previous test results (Failed: 1, Passed: 0), ordered by ascending age. Lists are delimited by [ ]. The implications of this conversion are important as it can help the previous works to re-assess their approaches and have more data for training and testing, as well as opening a broader data spectrum for future researchers in this field to find ready-to-use, rich datasets, on which they could evaluate their approaches and contribute to the TSP community. This also addresses the limitations in the field discussed in the systematic literature review <strong>[2]</strong>, stating that future research on TSP techniques should focus on collecting data from more recent subjects in a CI context with varying failure rates and larger execution times, as reproducible studies with appropriate datasets are needed to develop a usable body of knowledge regarding TSP over time. We believe that this conversion of the Westermo dataset is our contribution to alleviating the gap for the RL-based approaches. The original dataset can be found here.<strong>概述</strong> 本仓库包含经修改的近期发布的公开数据集Westermo的衍生版本。原始数据集采集自位于瑞典韦斯特罗斯的Westermo网络技术有限公司（Westermo Network Technologies AB），包含超过<strong>100万条测试判定结果</strong>，采集自连续<strong>500余天</strong>的夜间嵌入式系统测试过程。本衍生数据集经过改造与定制，专为科研社区打造，可用于解决回归测试选择、易波动测试（flaky tests）识别以及测试结果可视化等领域的难题。原始数据集可通过<strong>[1]</strong>中提及的参考文献获取。 Westermo数据集提供了关于测试用例执行过程及其对应结果的宝贵历史信息，可作为评估与对比不同测试用例选择与优先级排序（Test case Selection and Prioritization, TSP）技术的优质资源，帮助研究者识别后续执行中更易失败的测试用例。数据集中的测试用例具备执行时长、上次执行时间以及近期执行结果等属性。 Westermo数据集提供了关于测试用例执行过程及其对应结果的宝贵历史信息，可作为评估与对比不同测试用例选择与优先级排序（TSP）技术的优质资源，帮助研究者识别后续执行中更易失败的测试用例。数据集中的测试用例具备执行时长、上次执行时间以及近期执行结果等属性。 <strong>表1：数据集概览</strong> 测试用例数：1855 持续集成（Continuous Integration, CI）周期数：15197 测试判定总数：1036818 失败率：5.03% 然而，该数据集的特征多样性与丰富度对于部分TSP方法而言可能存在冗余。为此我们对数据集进行了转换改造，将Westermo数据集定制为与Paint Control和IOF/ROL这两类基于强化学习（Reinforcement Learning, RL）的TSP方法中广泛使用的数据集拥有相同特征的版本。本次转换需要合并多个原始变量并生成目标变量。在生成"LastResults"与"Cycle"字段值时，我们需要开展进一步分析，且数据处理环节需要深入理解夜间测试的执行流程。为此我们首先明确了该场景下CI周期的定义，并沿用了其对“会话（session）”的界定：“会话指在某一测试系统上，针对特定软件版本与测试工具版本运行整套测试用例的过程”。基于所使用的9个不同测试系统对数据进行拆分后，我们得到了9个适配CI场景的子数据集。 <strong>文件格式</strong> 压缩后的.zip文件包含9个数据文件，分别对应上述9个测试系统。数据集采用CSV格式存储，以分号（;）作为字段分隔符。下文的表格列出了数据集包含的所有字段及其说明。 <strong>表2：数据集字段参数</strong> jid：作业ID，与测试系统名称组合后，(jid, 系统名称)可作为测试会话的唯一主键 System：测试系统名称 Name：测试用例的唯一数字标识符 Verdict：本次测试执行的判定结果（失败：1，通过：0） Duration：测试用例的近似运行时长 Cycle：本次测试执行所属的CI周期编号 Group：测试用例所属的分组 LastRun：测试用例的上次执行时间，格式为日期时间字符串（格式：<em>YYYY-MM-DD HH:ii</em>） Id：测试执行的唯一数字标识符 CalcPrio：测试用例的优先级，由优先级排序算法计算得出（输出字段，初始值为0） result_array：按时间升序排列的过往测试结果列表（失败：1，通过：0），列表以方括号（[ ]）分隔 本次数据集转换具有重要意义：既可以帮助过往研究重新评估其方法，为训练与测试提供更多数据支持，也可为该领域的后续研究者提供现成的丰富数据集，帮助其验证自身方法并为TSP领域的发展贡献力量。同时，本转换也解决了<strong>[2]</strong>系统综述中提及的领域局限：该综述指出，未来TSP技术的研究应聚焦于从CI场景下的最新测试对象中采集数据，且需覆盖不同失败率与更长运行时长的测试用例——因为唯有通过合适的数据集开展可复现的研究，才能逐步构建起关于TSP的实用知识体系。我们认为，本次对Westermo数据集的转换，可为基于强化学习的TSP方法填补数据缺口做出贡献。原始数据集可在此处获取。