Dataset for the Paper: When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems

This dataset is associated with the paper titled "When Passing Tests Hides Vulnerabilities: An Empirical Study of Silent Failures in Agentic Systems". It is stored in a Microsoft Excel file that consists of five worksheets. Below is a simplified description of each worksheet. (1) The '01_Master' worksheet contains the 170 confirmed silent failures that form the study corpus, with one row per case. Each case records its iteration, identifiers (case_id, task_id), the agent framework and its architecture type (single- or multi-agent), the confirmed CWE and CWE family, the manual-review verdict (TP or Partial), and the qualitative codes assigned during analysis: the open code and axial category (Omission, Inadequacy, Introduction), the originator role, injection stage, propagation chain, code location, severity rating, and L3 exploitability score. All other worksheets are derived from this one. (2) The '02_RQ1_Taxonomy' worksheet reports the taxonomy and prevalence results (RQ1): the composition of the corpus by framework, architecture, and iteration; the distribution across the three axial categories; the ten open codes with their counts, shares, true-positive/partial split, and mean L3 score; and the overall true-positive versus partial breakdown. (3) The '03_RQ2_Propagation' worksheet reports the propagation and architecture results (RQ2): the artifact-level originator role, the causal origin of each propagation chain, the distribution of axial categories across single- and multi-agent architectures, and the distribution across code locations and injection stages. (4) The '04_RQ3_Vulnerability' worksheet reports the vulnerability-context results (RQ3): the distribution of silent failures across CWE families, severity levels, and L3 exploitability scores (summary statistics), together with the cross-tabulation of CWE family against axial category. (5) The '05_Convergence' worksheet reports the cross-framework convergence analysis: the (task, CWE) groups on which three or more frameworks independently produced a silent failure, listing for each group the number of frameworks, the number of cases, and the frameworks involved, along with the overall convergence rate. In addition to the spreadsheet, we provide the raw execution traces (raw_data_traces.zip). These are the normalized execution traces produced by the seven agent frameworks, including the generated patches and the outputs for each trace.