Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study

In this paper, we present the first measurement-driven study on the reliability of LLM-based expert systems applied to CTI tasks. We propose an automated framework, HalluVision, which generates LLM outputs, extracts malware-related entities using a fine-tuned Named Entity Recognition (NER) model, and evaluates their factual consistency using multiple similarity metrics. Our analysis, based on 4,940 real-world security articles, includes both quantitative measurements and qualitative case studies, offering a comprehensive evaluation of hallucination risks in this high-stakes application domain.