Investigating a NASA Cyclomatic Complexity Policy on Maintenance Risk of a Critical System

Monte is a mission critical system used by NASA for navigation and design of deep space missions has been used in over 40 missions over the last 18 years. Its continuous, reliable operation is considered critical to the operation for over 18 ongoing missions. Recently Monte has been escalated to safety-critical software and subject to NASA Software Assurance and Software Safety Standard requirements. One of these requirements mandates a policy that the cyclomatic complexity (CC) for safety-critical components be under 16 or given a technically detailed explanation as to why it cannot or should not be lower. Conformance to this requirement would be costly and we had doubts about its benefit and efficacy in managing our maintenance risk (defect proneness and defect repair effort). The requirement was not substantiated either empirically or in principle, and guidance in the literature for use of CC as in indicator of maintenance risk is limited and often speculative or have contradictory empirical or non-definitive results. This led us to rigorously investigate the impact the CC policy would have on the practical management of maintenance risk for Monte. The effect of CC on maintenance risk for Monte is explored using a variety of statistical methods and machine learning with aim of provide objective empirical evidence to support our decision as to what extent we will conform to the NASA CC policy of maintaining a CC under 16. This paper presents the conclusions and insights gained from this investigation. Practical questions related to the policy are addressed such as: Does higher CC associate with higher defect proneness? More effort to repair defects? If so, is there a CC after which action should be taken to reduce defect proneness and repair effort? We conclude that the impacts of CC on maintenance risk are, with high confidence, consistent with the risk management expectations of the NASA policy. We can quantify the benefit and weight it against the cost to inform our decision in conforming to the policy. Furthermore, we gained insight into how CC affects maintenance risk and is a useful indicator for maintenance risk management.