Replication Package: Lost in Configuration? Analyzing Misconfiguration Vulnerabilities with Large-Language Models

Welcome to the public repository for the additional content of the paper "Lost in Configuration? Analyzing Misconfiguration Vulenrabilities with Large-Language Models", submitted at the International Conference on Availability, Reliability and Security (ARES) 2026. This repository provides additional information to the empirical analysis of misconfiguration vulnerabilities fetched from the National Vulnerability Database, including the following files: PROMPTS.py: full prompting setup for ground-truth dataset and full dataset GROUND_TRUTH_ORIGINAL.jsonl: 60 CVEs with labeled ground truth data (15 CVEs per misconfiguration vulnerability type) GROUND_TRUTH_CLASSIFICATION_CSMODELS.jsonl: 60 CVEs of the ground truth, classified by four closed-source (CS) models GROUND_TRUTH_CLASSIFICATIONPROCESS_OSMODELS.csv: Classification process of the 60 CVEs of the ground truth, classified by 13 open-source (OS) models FULL_DATASET_CLASSIFICATION_6MODELS.jsonl: 4,231 CVEs, classified by 2 open-source and 4 closed-source models SELECTED_CVE_ANALYSIS.json: 2,491 selected CVEs of the full dataset, used for analyzing meta data, CWE, CVSS base metrics, and CVSS scores. README.txt: readMe file Keywords used for filtering (“*config*”) AND (“*misconfig*” OR “*wrong*” OR “*bad*” OR “*insecure*” OR “*default*” OR “*incorrect*” OR “*faulty*” OR “*unsafe*” OR “*poor*” OR “*flawed*” OR “*suboptimal*” OR “*missing*” OR “*deprecated*” OR “*noncompliant*” OR “*malicious*” OR “*insecure interaction*” OR “*improper*” OR “*weak default*” OR “*incorrect handling*” OR “*unprotected value*” OR “*invalid*” OR “*unauth*” OR “*insufficient*” OR “*uninitialized*” OR “*outdated*” OR “*hardcoded*” OR “*exposed*” OR “*debug*” OR “*unrestricted*”) Requirements Python >= 3.9; recommended Python packages for analysis: pandas, numpy, scikit-learn, krippendorff, matplotlib No further requirements License for using the data Creative Commons Attribution 4.0 International The Creative Commons Attribution license allows re-distribution and re-use of a licensed work on the condition that the creator is appropriately credited. Further information: https://creativecommons.org/licenses/by/4.0/legalcode