Data and Code for the paper "An Empirical Study on Exploratory Crowdtesting of Android Applications"

This package contains data and code to replicate the findings presented in our paper titled " <em>An Empirical Study on Exploratory Crowdtesting of Android Applications</em>". <strong>Abstract</strong> Crowdtesting is an emerging paradigm in which a ``crowd'' of people is recruited to perform testing tasks on demand. It proved to be especially promising in the mobile apps domain and in combination with exploratory testing strategies, in which individual testers pursue a creative, experience-based approach to design tests.<br> Managing the crowdtesting process, however, is still a challenging task, that can easily result either in wasteful spending or in inadequate software quality, due to the unpredictability of remote testing activities.<br> A number of works in the literature investigated the application of crowdtesting in the mobile apps domain. These works, however, investigated crowdtesting effectiveness in finding bugs, and not in scenarios in which the goal is to generate a re-executable test suite, as well. Moreover, less work has been conducted on to the impact of different exploratory testing strategies in the crowdtesting process.<br> As a first step towards filling this gap in the literature, in this work we conduct an empirical evaluation involving four open-source Android apps and twenty masters students, that we believe can be representative of practitioners partaking in crowdtesting activities. The students were asked to generate test suites for the apps using a Capture and Replay tool and different exploratory testing strategies. We then compare the effectiveness, in terms of aggregate code coverage, that different-sized crowds of students using different exploratory testing strategies may achieve. Results suggest that exploratory crowdtesting can be a valuable approach for generating GUI test suites for mobile apps, and provide a deeper insight on code coverage dynamics to project managers interested in using crowdtesting to test simple apps, on which they can make more informed decisions. <strong>Contents and Instructions</strong> This package contains: <strong>apps-under-test.zip</strong> A zip archive containing the source code of the four Android applications we considered in our study, namely MunchLife, TippyTipper, Trolly, and SimplyDo. <strong>InstrumentedSourceCode.zip</strong> A zip archive containing the instrumented source code of the four Android applications we used to compute branch coverage. <strong>students-test-suites.zip</strong> A zip archive containing the test suites developed by the students using Uninformed Exploratory Testing (referred to as "Black Box" in the subdirectories) and Informed Exploratory Testing (referred to as "White Box" in the subdirectories). This also includes coverage reports. <strong>compute-coverage-unions.zip </strong>A zip archive containing Python scripts we developed to compute the aggregate LOC coverage of all possible subsets of students. The scripts have been tested on MS Windows. To compute the LOC coverage achieved by any possible subsets of testers using IET and UET strategies, run the <em>analysisAndReport.py</em> script. To compute the LOC coverage achieved by mixed crowds in which some testers use a U+IET approach and others use a UET approach, run the <em>analysisAndReport_UET_IET_combinations_emma.py</em> script. <strong>branch-coverage-computation.zip </strong>A zip archive containing Python scripts we developed to compute the aggregate branch coverage of all considered subsets of students. The scripts have been tested on MS Windows. To compute the branch coverage achieved by any possible subsets of testers using UET and I+UET strategies, run the <em>branch_coverage_analysis.py</em> script. To compute the code coverage achieved by mixed crowds in which some testers use a U+IET approach and others use a UET approach, run the <em>mixed_branch_coverage_analysis.py</em> script. <strong>data-analysis-scripts.zip</strong> A zip archive containing R scripts to merge and manipulate coverage data, to carry out statistical analysis and draw plots. All data concerning RQ1 and RQ2 is available as a ready-to-use R data frame in the <em>./data/all_coverage_data.rds</em> file. All data concerning RQ3 is available in the <em>./data/all_mixed_coverage_data.rds </em>file.