Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study

In this paper, we present the first measurement-driven study on the reliability of LLM-based expert systems applied to CTI tasks. We propose an automated framework, HalluVision, which generates LLM outputs, extracts malware-related entities using a fine-tuned Named Entity Recognition (NER) model, and evaluates their factual consistency using multiple similarity metrics. Our analysis, based on 4,940 real-world security articles, includes both quantitative measurements and qualitative case studies, offering a comprehensive evaluation of hallucination risks in this high-stakes application domain.本文首次开展以实测为驱动的研究，针对应用于网络威胁情报（Cyber Threat Intelligence，CTI）任务的大语言模型（Large Language Model，LLM）专家系统的可靠性展开分析。本文提出自动化框架HalluVision，该框架可生成大语言模型输出结果，利用微调后的命名实体识别（Named Entity Recognition，NER）模型提取与恶意软件相关的实体，并通过多种相似度指标评估其事实一致性。本次研究基于4940篇真实安全文章展开，涵盖定量测量与定性案例研究两大维度，可为这一高风险应用领域内的大语言模型幻觉风险提供全面评估。