EMBERSim: A Large-Scale Databank for Boosting Similarity Search in Malware Analysis

In recent years there has been a shift from heuristics-based malware detection towards machine learning, which proves to be more robust in the current heavily adversarial threat landscape. While we acknowledge machine learning to be better equipped to mine for patterns in the increasingly high amounts of similar-looking files, we also note a remarkable scarcity of the data available for similarity-targeted research. Moreover, we observe that the focus in the few related works falls on quantifying similarity in malware, often overlooking the clean data. This one-sided quantification is especially dangerous in the context of detection bypass. We propose to address the deficiencies in the space of similarity research on binary files, starting from EMBER — one of the largest malware classification data sets. We enhance EMBER with similarity information as well as malware class tags, to enable further research in the similarity space. Our contribution is threefold: (1) we publish EMBERSim, an augmented version of EMBER, that includes similarity-informed tags; (2) we enrich EMBERSim with automatically determined malware class tags using the open-source tool AVClass on VirusTotal data and (3) we describe and share the implementation for our class scoring technique and leaf similarity method.

近年来，恶意软件检测领域已从基于启发式的检测转向机器学习方法，在当前对抗性威胁日益严峻的环境中，机器学习展现出更强的鲁棒性。尽管我们认可机器学习更适于从日益增多的相似外观文件中挖掘模式，但针对相似度研究的可用数据却极度匮乏。此外，现有少量相关研究多聚焦于量化恶意软件的相似度，却往往忽略了良性文件数据，这种片面的量化在检测绕过场景中尤为危险。针对二进制文件相似度研究领域存在的上述不足，我们以目前规模最大的恶意软件分类数据集之一EMBER为起点，提出了相应解决方案。我们为EMBER数据集补充相似度信息与恶意软件类别标签，以支撑相似度相关的后续研究。本研究的贡献主要包含三点：(1) 发布EMBER数据集的增强版EMBERSim，该版本纳入了基于相似度生成的标签；(2) 借助开源工具AVClass对VirusTotal平台的数据分析结果，为EMBERSim补充了自动生成的恶意软件类别标签；(3) 详细阐述并公开了我们的类别评分技术与叶相似度方法的实现代码。