Mobile Application Coverage: The 30% Curse and Ways Forward

Abstract Testing, security analysis, and other dynamic quality assurance approaches rely on mechanisms that invoke the software under test, aiming to achieve high code coverage. A large number of invocation mechanisms proposed in the literature, in particular for Android mobile applications, employ GUI-driven application exploration. However, studies show that even the most advanced GUI exploration techniques can cover only around 30% of a real-world application. This paper aims to investigate “the remaining 70%”. By conducting a large-scale experiment involving two human experts, who thoroughly explored 61 benchmark and 42 popular apps from Google Play, we show that achieving a substantially larger coverage for real-world applications is impractical even if we factor out known GUI-based exploration issues, such as the inability to provide semantic inputs and the right order of events. The main reasons preventing even human analysts from covering the entire application include application dependencies on remote servers and external resources, hard-to-reach app entry points, disabled and erroneous features, and software/hardware properties of the underlying device. Thus, future investment in GUI-based exploration strategies is unlikely to lead to substantial improvements in coverage. To chart possible ways forward and explore approaches to satisfy/bypass these “blockers”, we thoroughly analyze code-level properties guarding them. Our analysis shows that a large fraction of the blockers could actually be successfully bypassed with relatively simple beyond-GUI exploration techniques. We hope our study can inspire future work in this area; it also provides a realistic benchmark for evaluating such work. Purpose In this artifact, we provide our application exploration and coverage analysis results, in an effort to inspire the community to invest in coverage explainability and application-specific exploration strategies that go beyond GUI exploration. We include detailed information about our evaluation benchmarks for tool and manual exploration, e.g., package name, version used, source, category, downloads, SDK levels, permissions, etc. We also report coverage results achieved by tools and human analysts with plots of the coverage progression over time for analysts. We further provide manual analysis results for our case study, more specifically extracted reasons for unreachability of case study apps and associated code-level properties guarding unreached portions of the applications. Finally, we identify a list of beyond-GUI exploration techniques and categorize them for future work to reference and build upon when designing novel app exploration approaches. We hope for our results to serve as a precursor to building automated techniques which can identify and explain unreachable portions of an application from code, to constitute a ground truth for such techniques, and ultimately better orient testing/exploration efforts. Data from the Mobile Coverage Project, by Faridah Akinotcho, Lili Wei and Julia Rubin is made available under the Open Data Commons Attribution License v1.0: http://opendatacommons.org/licenses/by/1.0/