Widespread Error Detection in Large Scale Continuous Integration Systems

Dataset of 5000 json documents describing verification process of React project collected between 2023 and 2024. Errors recorded in this dataset were used in the presentation at CCIW workshop.

Abstract: Continuous Integration systems are widely used in the software industry to validate and integrate code changes into central repositories. Their effectiveness can be impacted by non-deterministic tests which can fail in the absence of any regression. Integration tests which depend on external services are particularly prone to this problem. We present a system which allows us to reduce the impact of non-deterministic failures by detecting widespread errors. The key assumption, which works well in practice, is that developers tend not to make identical mistakes simultaneously. If we observe a widespread error, it strongly suggests there is a problem with upstream services and not with the code change being evaluated. The detection algorithm consists of three main phases. First, the error text gets extracted from logs using predefined heuristics or automated methods. Then, this text gets fuzzy matched against a database of recently observed errors. Finally, statistics get checked to determine if they meet the criteria for a widespread error. When an error meets the criteria it either gets demoted to a warning or it gets enriched with information about an ongoing incident.