Exploring Rule Learning Algorithms for Detecting Faults on Larger Highly Configurable Systems: The axTLS Project

Highly configurable systems are software systems that can be configured in a large number of ways to meet different user requirements. Testing such systems is particularly challenging due to the high number of possible configurations, making exhaustive testing infeasible. This paper builds upon state of the art approaches by evaluating an expanded set of rule learning algorithms on a case study with a larger set of features. Specifically, we focus on Kconfig based systems, which represent a dominant category of highly configurable systems in practice. We use multivariate statistical analysis to assess the performance of these algorithms across multiple performance metrics. Our findings reveal that certain algorithms, such as JRip and PART, C5.0 perform higher performance measures and interpretability, making them a suitable approach for fault detection in larger, highly configurable systems. This dataset provides all the code and data that we used in this paper.