Causality Research using Interventions Like Events

Abstract—Context Intervention experiments are consideredthe gold standard in causality research. When conducting anintervention requires too much effort, it limits the number ofinterventions. This leads to a trade-off between the cleanlinessof interventions and large robust datasets. Aim We presenta method to identify natural events similar to interventions,building large intervention datasets. These similar events canallow us to investigate causality on a large scale. To achieve theobjective, we profile the interventions using labeling functions,validate them, and investigate the impact of intervention-likeevents. Method We demonstrate our method in the domain ofsoftware quality improvement. Static analyzers are implicitlybuilt on the premise that fixing their alerts should improvequality. We quantify the impact of such alert removals. First,we built a dataset of 521 manual alert-removing interventions,allowing us to learn how the interventions look. We representthe intervention profile in the labeling functions. We appliedthese labeling functions to code commits, found intervention-likenatural events, and used them to analyze the impact of the alertremoval. This resulted in a much larger dataset that had 8,245alert removals, more than 15 times larger than our dataset ofmanual interventions. The scale of the dataset enables evaluatingthe impact of interventions. Using removals done in live projectsallowed us to measure the impact on the tendency to bugs. ResultsWe identified complexity-reducing interventions that reduce theprobability of future bugs. Such interventions are relevant to 33%of Python files and can reduce the tendency to bugs by up to 5.5percentage points. Conclusions Using our method saved 475 daysneeded to build the dataset manually. This allowed us to evaluateinterventions of various types on a large scale and to evaluatetheir causal impact. The method provides a large number ofnatural interventions that are highly needed in causality researchin many domains.