Replication Package for the paper "Web App Performance Testing in Industrial Contexts: Supporting Workload Generation with E2E-Loader++"

Replication package for the JSS paper "Web App Performance Testing in Industrial Contexts: Supporting Workload Generation with E2E-Loader++" Content This replication package includes: An Excel file (subject_group_assignment.xlsx) detailing the experimental design we adopted and the assignment of different subject to testing tasks. Test artifacts generated by practitioners manually and using E2E-Loader++ (limited to the open-source application involved in our study) Test execution data obtained by executing the performance tests generated by practitioners Anonymized results of the post-task survey we conducted Analysis scripts that can be used to replicate our analyses, results, and data visualizations. Test Artifacts Test artifacts are available in the test_artifacts directory. The directory contains a subdirectory for each of the six practitioners involved in our study (S1 to S6). Each subdirectory contains the two performance tests produced by that specific subject. The performance tests adopt the following convention: {Subject-ID}_{Session-ID}_{App-ID}_{Approach}.jmx, where: {Subject-ID} is a value in S1-S6 identifying the subject. {Session-ID} is a value in {A, B} and identifies in which experimental session that performance test was produced. {App-ID} identifies the system for which that test was produced. The public dataset only includes tests generated for App3, which is the open source app. {Approach} is a value in {manual, tool}, with manual indicating that the manual generation approach was used for that artifact, while tool indicates that E2E-Loader was used to generate that performance test. In addition, a gold-standard directory is provided as well, containing the reference workload employed in our study for App 3. Test Execution Data Test execution data is provided in the directories caleido-cpu-usage-logs and booker-cpu-usage-logs for App 1 and App 3, respectively. The files in the above directories adopt the following naming convention: cpu_usage_log_{Subject-ID}_{Session-ID}_{Approach}_{Repetition-ID}.csv, where: {Subject-ID} is a value in S1-S6 identifying the subject. {Session-ID} is a value in {A, B} and identifies in which experimental session that performance test was produced. {Approach} is a value in {M, E}, with M indicating that the recorded load was induced by a workload generated manually, E indicates that E2E-Loader++ was used. {Repetition-ID} is a value in {1,2,3}, identifying in which of the three repetitions of the performance test execution that load was collected. In addition, data from the execution of the reference workloads is included as well, in files named cpu_usage_log_gold_{Repetition-ID}.csv Results and Scripts RQ1 The results for RQ1 are provided in the Excel file RQ1-results.xlsx. The Excel file includes a table with all the investigated usage scenarios for each of the three considered applications, as well as the number of True Positives, False Positives and False Negatives detected in each scenario. Last, the spreadsheet includes formular to compute the Precision, Recall and F1 metrics. RQ2 The results for RQ1 are provided in the Excel file RQ2-results.xlsx. To replicate these results, it is necessary to run the RQ2-analysis.R script. RQ3 The raw data collected during our experiments to answer RQ3 is available in the Excel file RQ3-data.xlsx. The data includes time employed by each subject to complete the testing tasks under the different conditions, and number of interactions (clicks, keystrokes, scrolls) required. Results can be replicated by running the RQ3.R scripts. RQ4 The raw data collected via our post-task survey is in the RQ4-data.xlsx file. Our results can be replicated by running the RQ4-analysis.R file.