AI Agents in Support of Software Design and Architecture: Pilot Calibration Package for the Systematic Literature Review Protocol

Calibration package for the systematic literature review protocol onLLM-based agents in software design and architecture. Six reviewers independently screened the same 30 records, selected toinclude borderline cases, applying the inclusion and exclusion criteriaand the operational definition of an agent. Agreement was measured withFleiss' kappa across the six reviewers (0.691; 95% CI 0.459-0.854) andCohen's kappa per pair. Twenty-one of the thirty records were decidedunanimously. The threshold defined a priori in the protocol was 0.60. The package contains the six completed screening forms, the blank form,the consolidated decision matrix, the agreement-analysis script and itsoutput, the list of the 30 pilot records (identifiers and metadata only,without abstracts), and version 1.0 of the Coding Guide derived from theconsensus meeting. To recompute the agreement: python3 analise_kappa.py ./fichas Reviewers are identified by code (R1-R6). The screening forms aredeposited exactly as applied between 27 and 30 July 2026.