Replication package for "Leveraging Large Language Models for Automated Diagnosis of CI/CD Pipeline Failures"

Replication package for the M.Sc. thesis "Leveraging Large Language Models forAutomated Diagnosis of CI/CD Pipeline Failures" (Tampere University, 2026). The study evaluated three conditions on identical preprocessed input from 131failed GitHub Actions runs in the scrapy/scrapy and vitejs/vite repositories: adeterministic rule-based baseline, a locally hosted open-weights model, and ahosted proprietary model. This package contains the evidence behind the held-outresults. Contents: the full set of 53 human annotations, each with a failure category, awritten root cause and the verified evidence line numbers; the collection, triageand partition manifests; the freeze record fixing the prompt, system message,response schema, taxonomy, preprocessing and decoding settings before theheld-out run; the annotation codebook; per-case predictions for all threeconditions over the 31 held-out cases; the per-case preprocessing trace; thepaired statistical report; and the results organised by research question. A verification script checks every file against its recorded SHA-256 digest andthen confirms the provenance chain: that the annotation set and partitionmanifest match the freeze record, that the held-out run scored the frozenannotations under the frozen prompt, and that it executed at the frozen sourcecommit. The GitHub Actions logs the annotations describe belong to the scrapy/scrapy andvitejs/vite projects and are not redistributed here. The package publishes theirSHA-256 digests instead, so a reader holding the same archives can confirm theyare the same logs. GitHub expires Actions logs after a retention period, so thecollected cohort cannot be retrieved again; independent replication is bounded bythe stored per-case records. Publication of the full annotation set with checksums was one of the mitigationsrecorded with the supervisory approval of the single-annotator ground-truthdesign.