SereneWAve/LAMDA

--- license: mit size_categories: - 1M<n<10M tags: - android - malware - cybersecurity - concept-drift - longitudinal - security - classification - benchmark configs: - config_name: Baseline data_files: - split: train path: Baseline/*/*_train.parquet - split: test path: Baseline/*/*_test.parquet # - config_name: var_thresh_0.0001 # data_files: # - split: train # path: var_thresh_0.0001/*/*_train.parquet # - split: test # path: var_thresh_0.0001/*/*_test.parquet - config_name: var_thresh_0.01 data_files: - split: train path: var_thresh_0.01/*/*_train.parquet - split: test path: var_thresh_0.01/*/*_test.parquet language: - en pretty_name: LAMDA --- # LAMDA: A Longitudinal Android Malware Dataset for Drift Analysis This dataset contains a longitudinal benchmark for Android malware detection designed to analyze and evaluate concept drift in machine learning models. It includes labeled and feature-engineered Android APK data from 2013 to 2025 (excluding 2015), with over 1 million samples collected from real-world sources. ## Dataset Details ### Dataset Description - **Curated by:** IQSeC Lab, The University of Texas at El Paso - **Shared by:** IQSeC Lab - **Language(s):** Not applicable - **License:** MIT LAMDA is the largest and most temporally diverse Android malware dataset to date. It enables studies in concept drift, temporal generalization, family classification, and robust malware detection over time. Each sample includes static features (from `.data` files), metadata (VirusTotal detections, family name, timestamp), and binary labels. The dataset was created using ~1M APKs from [AndroZoo](https://androzoo.uni.lu/), with additional metadata and labels derived via VirusTotal and AVClass2. Labels are assigned using a 4+ AV detection threshold to reduce noise. ### Dataset Sources - **Repository:** https://huggingface.co/datasets/IQSeC-Lab/LAMDA - **Project Website:** https://iqsec-lab.github.io/LAMDA/ - **Paper:** https://arxiv.org/abs/2505.18551 ## Uses ### Direct Use - Malware classification - Family prediction - Concept drift analysis - Temporal generalization benchmarks - SHAP-based feature attribution drift analysis - Continual learning evaluation (e.g., class-IL, replay) ### Out-of-Scope Use - Dynamic behavior analysis (no runtime traces) - On-device malware detection (model integration not provided) ## Dataset Structure Each year is stored in a subdirectory: ``` 2013/ ├── 2013_train.parquet ├── 2013_test.parquet ... 2025/ ├── 2025_train.parquet ├── 2025_test.parquet ``` Each `.parquet` contains: | Column | Description | |---------------|-----------------------------------------------------| | `label` | 0 = benign, 1 = malware | | `family` | Malware family name (via AVClass2) | | `vt_count` | VirusTotal vendor detection count | | `year_month` | Timestamp in YYYY-MM format | | `feat_0 ... feat_4560` | Static bag-of-words features (int8) | | `hash` | Sample SHA256 hash (used as index) | A `feature_mapping.csv` maps each `feat_i` to its original static token. ## Dataset Creation ### Curation Rationale To enable longitudinal and realistic evaluation of ML-based malware detection systems that must remain effective in the face of temporal and adversarial drift. ### Source Data APK samples were downloaded from AndroZoo and processed using static analysis to extract `.data` files. Metadata was merged from a curated CSV containing VirusTotal counts and family assignments via AVClass2. #### Data Collection and Processing - Extracted feature vectors from `.data` files (comma-separated tokens) - Labeled malware if `vt_detection ≥ 4` - Assigned families via AVClass2 - Feature vectors vectorized using bag-of-words (sparse) - Feature selection via `VarianceThreshold=0.001` → 4,561 features - Train/test split (80/20) stratified by label, year-wise #### Who are the source data producers? Original APKs are from AndroZoo. Annotations and processing were conducted by IQSeC Lab at the University of Texas at El Paso. ### Annotations #### Annotation Process - Malware/benign labels based on AV vendor threshold (≥4) - Family labels from AVClass2 - All annotations generated using automated pipelines #### Who are the annotators? Researchers at IQSeC Lab via static tooling and AVClass2 #### Personal and Sensitive Information No PII or private user data is included. APKs are anonymized binaries. ## Bias, Risks, and Limitations - Biased toward highly detected malware (AV-centric labeling) - No dynamic/runtime behavior - Concept drift is dataset-driven, not simulation-based ### Recommendations - Normalize class balance before training - Use continual or time-aware validation schemes - SHAP explanations should be anchored year-wise ## Citation **BibTeX:** ```bibtex @inproceedings{haque2026lamda, title = {LAMDA: A Longitudinal Android Malware Benchmark for Concept Drift Analysis}, author = {Haque, Md Ahsanul and Hossain, Ismail and Kamol, Md Mahmuduzzaman and Alam, Md Jahangir and Amalapuram, Suresh Kumar and Talukder, Sajedul and Rahman, Mohammad Saidur}, booktitle = {The 14th International Conference on Learning Representations (ICLR)}, year = {2026}, url = {https://openreview.net/forum?id=1FnCrZtBNQ} }license: MIT许可证 size_categories: - 100万<n<1000万 tags: - 安卓（Android） - 恶意软件（malware） - 网络安全（cybersecurity） - 概念漂移（concept-drift） - 纵向（longitudinal） - 安全（security） - 分类（classification） - 基准数据集（benchmark） configs: - config_name: 基线（Baseline） data_files: - split: 训练集（train） path: Baseline/*/*_train.parquet - split: 测试集（test） path: Baseline/*/*_test.parquet # - config_name: 方差阈值0.0001（var_thresh_0.0001） # data_files: # - split: 训练集（train） # path: var_thresh_0.0001/*/*_train.parquet # # - split: 测试集（test） # path: var_thresh_0.0001/*/*_test.parquet - config_name: 方差阈值0.01（var_thresh_0.01） data_files: - split: 训练集（train） path: var_thresh_0.01/*/*_train.parquet - split: 测试集（test） path: var_thresh_0.01/*/*_test.parquet language: - en pretty_name: LAMDA # LAMDA：用于漂移分析的纵向安卓恶意软件数据集 本数据集为安卓恶意软件检测任务提供纵向基准，旨在分析与评估机器学习模型中的概念漂移现象。数据集涵盖2013年至2025年（不含2015年）的带标注且经特征工程处理的安卓APK数据，包含超100万条来自真实场景的样本。 ## 数据集详情 ### 数据集描述 - **整理方：** 得克萨斯大学埃尔帕索分校IQSeC实验室 - **共享方：** IQSeC实验室 - **语言：** 不适用 - **许可证：** MIT许可证 LAMDA是目前规模最大、时间维度最丰富的安卓恶意软件数据集，可支撑概念漂移、时间泛化、恶意软件家族分类以及长期鲁棒恶意软件检测等方向的研究。每条样本包含静态特征（源自`.data`文件）、元数据（VirusTotal检测结果、恶意软件家族名称、时间戳）以及二分类标签。 本数据集基于AndroZoo平台的约100万条APK样本构建，额外元数据与标签通过VirusTotal与AVClass2生成。为降低噪声，标签采用“≥4款杀毒软件（AV）检测为恶意”的阈值进行标注。 ### 数据集来源 - **仓库地址：** https://huggingface.co/datasets/IQSeC-Lab/LAMDA - **项目官网：** https://iqsec-lab.github.io/LAMDA/ - **相关论文：** https://arxiv.org/abs/2505.18551 ## 适用场景 ### 直接使用场景 - 恶意软件分类 - 恶意软件家族预测 - 概念漂移分析 - 时间泛化基准测试 - 基于SHAP的特征归因漂移分析 - 持续学习评估（如类别增量学习、重放式学习） ### 不适用场景 - 动态行为分析（无运行时轨迹数据） - 端侧恶意软件检测（未提供模型集成方案） ## 数据集结构 按年份存储为子目录： 2013/ ├── 2013_train.parquet ├── 2013_test.parquet ... 2025/ ├── 2025_train.parquet ├── 2025_test.parquet 每个Parquet格式文件（parquet）包含以下列： | 列名 | 描述 | |---------------|-----------------------------------------------------| | `label` | 0 = 良性软件（benign），1 = 恶意软件（malware） | | `family` | 恶意软件家族名称（通过AVClass2生成） | | `vt_count` | VirusTotal检测到的杀毒厂商数量 | | `year_month` | 格式为YYYY-MM的时间戳 | | `feat_0 ... feat_4560` | 静态词袋特征（int8类型） | | `hash` | 样本的SHA256哈希值（用作索引） | 另有`feature_mapping.csv`文件，用于将每个`feat_i`映射至其对应的原始静态Token（token）。 ## 数据集构建 ### 构建初衷 旨在对基于机器学习的恶意软件检测系统进行纵向且贴合真实场景的评估，此类系统需在面对时间漂移与对抗性漂移时仍保持有效性。 ### 源数据 APK样本从AndroZoo平台下载，通过静态分析提取`.data`文件。元数据来自整理后的CSV文件，包含VirusTotal检测结果与通过AVClass2生成的恶意软件家族分配信息。 #### 数据收集与处理流程 - 从`.data`文件（以逗号分隔的Token）中提取特征向量 - 当`vt_detection ≥4`时，将样本标注为恶意软件 - 通过AVClass2分配恶意软件家族标签 - 采用词袋模型（稀疏表示）对特征向量进行向量化 - 通过`VarianceThreshold=0.001`进行特征选择，最终保留4561个特征 - 按标签与年份进行分层，以80:20的比例划分为训练集与测试集 #### 源数据生产者 原始APK样本来自AndroZoo平台，标注与处理工作由得克萨斯大学埃尔帕索分校的IQSeC实验室完成。 ### 标注信息 #### 标注流程 - 恶意/良性标签基于杀毒厂商检测阈值（≥4）生成 - 家族标签来自AVClass2 - 所有标注均通过自动化流水线生成 #### 标注者 IQSeC实验室的研究人员通过静态分析工具与AVClass2完成标注 #### 个人与敏感信息 数据集未包含任何个人可识别信息（PII）或用户隐私数据，所有APK均为匿名化二进制文件。 ## 偏差、风险与局限性 - 偏向高检测率的恶意软件（基于杀毒软件的标注方式存在偏差） - 未包含动态/运行时行为数据 - 概念漂移由数据集本身的时间分布驱动，而非模拟生成 ### 建议 - 训练前对类别分布进行均衡化处理 - 采用持续学习或时间感知的验证策略 - SHAP解释需按年份进行锚定分析 ## 引用信息 **BibTeX格式引用：** bibtex @inproceedings{haque2026lamda, title = {LAMDA: A Longitudinal Android Malware Benchmark for Concept Drift Analysis}, author = {Haque, Md Ahsanul and Hossain, Ismail and Kamol, Md Mahmuduzzaman and Alam, Md Jahangir and Amalapuram, Suresh Kumar and Talukder, Sajedul and Rahman, Mohammad Saidur}, booktitle = {The 14th International Conference on Learning Representations (ICLR)}, year = {2026}, url = {https://openreview.net/forum?id=1FnCrZtBNQ} }