it4lia/EMBER_cleaned
收藏资源简介：
--- language: - en license: mit pretty_name: EMBER Cleaned task_categories: - tabular-classification tags: - cybersecurity - malware - static-analysis - pe-files - malware-detection - benchmark - tabular - ai-ready - clustering size_categories: - 1M<n<10M --- # EMBER Cleaned EMBER Cleaned is a cleaned and AI-ready version of the original **EMBER (Endgame Malware Benchmark for Research)** dataset, a widely used benchmark for static malware detection on Windows Portable Executable (PE) files. The original EMBER dataset was introduced by Endgame / Elastic as an open benchmark for machine-learning-based malware detection using only **static PE-derived features**, without executing binaries. This cleaned release preserves that purpose while making the dataset easier to load, more reproducible, and more directly usable for downstream experimentation. Compared with the original source asset, this release standardizes metadata, removes duplicate samples, drops constant features, and exports unlabeled samples into a dedicated split for semi-supervised workflows. Each sample is represented as a fixed-length numerical vector derived from PE structure and content, including header information, section statistics, imports, and histogram-based features. ## Original dataset This dataset is a cleaned derivative of the original EMBER benchmark: - **Original name:** EMBER (Endgame Malware Benchmark for Research) - **Original provider:** Endgame / Elastic - **Original paper:** Anderson, H. S., & Roth, P. (2018). *EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models* - **Original DOI:** https://doi.org/10.48550/arXiv.1804.04637 - **Original project/repository:** https://github.com/elastic/ember Please cite the original EMBER paper when using this cleaned release in research. ## Files | File | Description | |---|---| | `ember_clean.npz` | Index file with row/feature counts and file references | | `ember_clean_X.npy` | Feature matrix (`float32`), raw memmap, shape `(799838, 2099)` | | `ember_clean_y.npy` | Label vector (`int32`), `0 = benign`, `1 = malware` | | `ember_clean_metadata.parquet` | Per-sample metadata: SHA-256, timestamps, malware-related fields when available, quality flags | | `ember_unlabeled.npz` | Index file for unlabeled split | | `ember_unlabeled_X.npy` | Unlabeled feature matrix with the same 2,099 features | | `ember_unlabeled_y.npy` | Label marker array (`int32`) for the unlabeled split; expected to contain only `-1` values | | `ember_clean_metadata_unlabeled.parquet` | Metadata for unlabeled samples | | `manifest.json` | Versioned manifest with checksums and artifact references | | `ember_cleaned_dataset.ipynb` | Exploration and usage notebook | ## What’s in the dataset? This cleaned release contains the **labeled portion** of EMBER plus a **separate unlabeled split**. ### Labeled split - **799,838 labeled samples** - approximately balanced between benign and malicious files - **2,099 numerical features** - feature dtype: `float32` - label dtype: `int32` - labels: - `0` = benign - `1` = malware ### Unlabeled split - **199,966 unlabeled samples** - exported separately for semi-supervised workflows - same 2,099-dimensional feature space - not intended to be interpreted as benign or malicious ground truth ### Feature representation Samples are not raw executables. Each file is represented as a **fixed-length static feature vector** extracted from the original PE file. These features describe structural and statistical properties of the binary, such as: - PE headers - imported APIs / libraries - sections - byte-histogram-related information - entropy-related characteristics ## Cleaning summary This release is the output of a quality-control and standardization pipeline applied to the original EMBER artifacts. Main processing steps: 1. **Duplicate removal** using feature fingerprints 2. **Constant-feature filtering**, reducing the feature space from 2,381 to **2,099** 3. **Metadata standardization** 4. **Missing-value normalization and quality flagging** 5. **Label separation**, exporting `label = -1` samples into a dedicated unlabeled split 6. **Manifest generation** for reproducibility and integrity checks Summary of the main changes: - **196 duplicate samples removed** - **282 constant features dropped** - **199,966 unlabeled samples exported separately** - final labeled dataset shape: **799,838 × 2,099** ## File structure ```text EMBER_cleaned/ ├── ember_clean.npz ├── ember_clean_X.npy ├── ember_clean_y.npy ├── ember_clean_metadata.parquet ├── ember_unlabeled.npz ├── ember_unlabeled_X.npy ├── ember_unlabeled_y.npy ├── ember_clean_metadata_unlabeled.parquet └── manifest.json ``` The .npz index stores _rows and _features for reliable loading. The feature matrices are raw memmap-backed arrays and should be loaded with explicit dtype and shape. ## Requirements To run the quickstart examples, install the minimum required dependencies: ```bash pip install numpy pandas pyarrow ``` For notebook-based exploration and basic visualization, you may also install: ```bash pip install jupyter matplotlib seaborn scikit-learn ``` ## Quickstart This example loads the labeled EMBER Cleaned split and checks that features, labels, and metadata are consistent and ready for supervised use. ```python import numpy as np import pandas as pd idx = np.load("ember_clean.npz", allow_pickle=True) n_rows = int(idx["_rows"]) n_features = int(idx["_features"]) X = np.fromfile("ember_clean_X.npy", dtype=np.float32).reshape(n_rows, n_features) y = np.load("ember_clean_y.npy") meta = pd.read_parquet("ember_clean_metadata.parquet") print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features | " f"labels: {y.shape[0]} | " f"metadata columns: {meta.shape[1]}") assert X.shape[0] == len(y) == len(meta) assert set(np.unique(y)) == {0, 1} print("Unique labels:", np.unique(y)) print("Labeled Metadata Columns:", meta.columns.tolist()) print("All checks passed.") ``` The following example loads the unlabeled EMBER Cleaned split and checks that features and metadata are aligned for semi-supervised or exploratory use. ```python import numpy as np import pandas as pd idx_u = np.load("ember_unlabeled.npz", allow_pickle=True) n_rows_u = int(idx_u["_rows"]) n_features_u = int(idx_u["_features"]) X_u = np.fromfile("ember_unlabeled_X.npy", dtype=np.float32) assert X_u.size == n_rows_u * n_features_u, ( f"Unexpected X_u size: got {X_u.size}, expected {n_rows_u * n_features_u}" ) X_u = X_u.reshape(n_rows_u, n_features_u) meta_u = pd.read_parquet("ember_clean_metadata_unlabeled.parquet") print(f"Dataset: {X_u.shape[0]} samples, {X_u.shape[1]} features | " f" unlabeled split: {n_rows_u} samples | " f"metadata columns: {meta_u.shape[1]}") assert X_u.shape == (n_rows_u, n_features_u) assert len(meta_u) == n_rows_u if "label_int" in meta_u.columns: print("Unlabeled metadata labels:", np.unique(meta_u["label_int"])) assert set(np.unique(meta_u["label_int"])) == {-1} print("Unlabeled metadata columns:", meta_u.columns.tolist()) print("Unlabeled split loaded successfully.") ``` ## Notebook The repository also includes an exploration notebook in `.ipynb` format, designed to provide additional context on the cleaned dataset, its structure, and its main analytical use cases. The notebook can be used to: - inspect the labeled and unlabeled splits - explore metadata fields and label distributions - validate dataset consistency - review example analyses and downstream use cases To open it locally, run: ```bash jupyter notebook ember_cleaned_dataset.ipynb ``` or, if you use JupyterLab: ```bash jupyter lab ember_cleaned_dataset.ipynb ``` Make sure to open the notebook from the dataset root directory so that relative file paths resolve correctly. ## Typical use cases EMBER Cleaned supports: - binary malware detection - benchmarking of tabular ML pipelines - feature importance analysis - semi-supervised learning using the separate unlabeled split - exploratory data analysis - representation learning and clustering The accompanying notebook includes dataset loading, exploratory analysis, and example use cases focused on discriminative features and model evaluation. ## Notes and limitations This is a static-analysis dataset only. The cleaned release contains derived features, not raw PE binaries. The unlabeled split should not be treated as ground truth. Results on EMBER should not be over-generalized to modern malware without additional validation. The dataset is intended for defensive research, benchmarking, and education. ## License This cleaned release is derived from EMBER. The original EMBER data files are associated with the MIT License. Please verify that your downstream redistribution and reuse remain aligned with the original EMBER terms. ## References If you use this dataset, please cite the original EMBER paper: @article{anderson2018ember, title={EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models}, author={Anderson, Hyrum S. and Roth, Phil}, journal={arXiv preprint arXiv:1804.04637}, year={2018} } ## APA: Anderson, H. S., & Roth, P. (2018). EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. arXiv. https://doi.org/10.48550/arXiv.1804.04637 ## Contacts Shared by: ACN
language: - en license: mit pretty_name: EMBER Cleaned task_categories: - tabular-classification tags: - cybersecurity - malware - static-analysis - pe-files - malware-detection - benchmark - tabular - ai-ready - clustering size_categories: - 1M<n<10M --- # EMBER Cleaned数据集 EMBER Cleaned 是原始**EMBER（Endgame Malware Benchmark for Research，研究用恶意软件基准数据集）**数据集的净化版且适配AI的版本，该数据集是针对Windows可移植可执行（Portable Executable，PE）文件开展静态恶意软件检测的广泛使用的基准数据集。 原始EMBER数据集由Endgame/Elastic推出，是一款仅使用**静态PE衍生特征**且无需执行二进制文件的、基于机器学习的恶意软件检测开源基准。本次净化版保留了该核心目标，同时优化了数据集的加载便捷性、可复现性，使其更可直接用于下游实验研究。 相较于原始源数据集，本次净化版实现了元数据标准化，移除了重复样本，剔除了恒定特征，并将未标记样本导出至专属划分集以支持半监督工作流。每个样本均表示为源自PE文件结构与内容的定长数值向量，涵盖头部信息、节区统计量、导入表信息以及基于直方图的特征。 ## 原始数据集 本数据集是原始EMBER基准的净化衍生版本： - **原始名称**：EMBER（Endgame Malware Benchmark for Research，研究用恶意软件基准数据集） - **原始提供方**：Endgame/Elastic - **原始文献**：Anderson, H. S. 与 Roth, P. (2018). *EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models* - **原始DOI**：https://doi.org/10.48550/arXiv.1804.04637 - **原始项目/仓库**：https://github.com/elastic/ember 若在研究中使用本净化版数据集，请引用原始EMBER文献。 ## 文件说明 | 文件名称 | 描述 | |---|---| | `ember_clean.npz` | 存储行/特征数量与文件引用的索引文件 | | `ember_clean_X.npy` | 特征矩阵（`float32`类型），原生内存映射（memmap）格式，形状为`(799838, 2099)` | | `ember_clean_y.npy` | 标签向量（`int32`类型），`0`表示良性软件，`1`表示恶意软件 | | `ember_clean_metadata.parquet` | 单样本元数据：包含SHA-256哈希、时间戳、可用的恶意软件相关字段与质量标记 | | `ember_unlabeled.npz` | 未标记划分集的索引文件 | | `ember_unlabeled_X.npy` | 未标记特征矩阵，包含与标记集一致的2099个特征 | | `ember_unlabeled_y.npy` | 未标记划分集的标签标记数组（`int32`类型），预期仅包含`-1`值 | | `ember_clean_metadata_unlabeled.parquet` | 未标记样本的元数据 | | `manifest.json` | 包含校验和与制品引用的版本化清单文件 | | `ember_cleaned_dataset.ipynb` | 数据集探索与使用指南笔记本 | ## 数据集内容 本净化版数据集包含EMBER的**标记部分**与**独立的未标记划分集**。 ### 标记划分集 - **799,838个标记样本** - 良性与恶意样本数量近似均衡 - **2099个数值型特征** - 特征数据类型：`float32` - 标签数据类型：`int32` - 标签定义： - `0` = 良性软件 - `1` = 恶意软件 ### 未标记划分集 - **199,966个未标记样本** - 单独导出以支持半监督工作流 - 特征空间维度与标记集一致，为2099维 - 不可将其视为良性/恶意的真实标签 ### 特征表示形式 本数据集并非原始可执行文件，每个文件均表示为从原始PE文件中提取的**定长静态特征向量**。此类特征描述了二进制文件的结构与统计特性，例如： - PE头部信息 - 导入的应用程序编程接口（API，Application Programming Interface）与库 - PE节区 - 字节直方图相关信息 - 熵相关特征 ## 净化流程说明 本净化版是对原始EMBER数据集制品执行质量控制与标准化流程后的产物。 主要处理步骤： 1. **重复样本移除**：基于特征指纹识别并移除重复样本 2. **恒定特征过滤**：将特征空间从2381维缩减至**2099维** 3. **元数据标准化**：统一元数据格式与字段 4. **缺失值归一化与质量标记**：处理缺失值并添加质量标记 5. **标签分离**：将`label = -1`的样本导出至专属未标记划分集 6. **生成校验清单**：用于保障可复现性与完整性校验 主要变更汇总： - **移除196个重复样本** - **剔除282个恒定特征** - **单独导出199,966个未标记样本** - 最终标记数据集形状：**799,838 × 2,099** ## 文件目录结构 text EMBER_cleaned/ ├── ember_clean.npz ├── ember_clean_X.npy ├── ember_clean_y.npy ├── ember_clean_metadata.parquet ├── ember_unlabeled.npz ├── ember_unlabeled_X.npy ├── ember_unlabeled_y.npy ├── ember_clean_metadata_unlabeled.parquet └── manifest.json 其中.npz索引文件存储了`_rows`与`_features`信息，以确保可靠加载。特征矩阵为原生内存映射（memmap）数组，加载时需显式指定数据类型与形状。 ## 依赖要求 若需运行快速入门示例，请安装最低依赖： bash pip install numpy pandas pyarrow 若需基于Jupyter笔记本开展探索与基础可视化，还可安装： bash pip install jupyter matplotlib seaborn scikit-learn ## 快速入门 本示例将加载标记版EMBER Cleaned数据集，并验证特征、标签与元数据的一致性，确保其可用于监督学习任务。 python import numpy as np import pandas as pd idx = np.load("ember_clean.npz", allow_pickle=True) n_rows = int(idx["_rows"]) n_features = int(idx["_features"]) X = np.fromfile("ember_clean_X.npy", dtype=np.float32).reshape(n_rows, n_features) y = np.load("ember_clean_y.npy") meta = pd.read_parquet("ember_clean_metadata.parquet") print(f"Dataset: {X.shape[0]} samples, {X.shape[1]} features | " f"labels: {y.shape[0]} | " f"metadata columns: {meta.shape[1]}") assert X.shape[0] == len(y) == len(meta) assert set(np.unique(y)) == {0, 1} print("Unique labels:", np.unique(y)) print("Labeled Metadata Columns:", meta.columns.tolist()) print("All checks passed.") 以下示例将加载未标记版EMBER Cleaned数据集，并验证特征与元数据的对齐性，以支持半监督或探索性任务。 python import numpy as np import pandas as pd idx_u = np.load("ember_unlabeled.npz", allow_pickle=True) n_rows_u = int(idx_u["_rows"]) n_features_u = int(idx_u["_features"]) X_u = np.fromfile("ember_unlabeled_X.npy", dtype=np.float32) assert X_u.size == n_rows_u * n_features_u, ( f"Unexpected X_u size: got {X_u.size}, expected {n_rows_u * n_features_u}" ) X_u = X_u.reshape(n_rows_u, n_features_u) meta_u = pd.read_parquet("ember_clean_metadata_unlabeled.parquet") print(f"Dataset: {X_u.shape[0]} samples, {X_u.shape[1]} features | " f" unlabeled split: {n_rows_u} samples | " f"metadata columns: {meta_u.shape[1]}") assert X_u.shape == (n_rows_u, n_features_u) assert len(meta_u) == n_rows_u if "label_int" in meta_u.columns: print("Unlabeled metadata labels:", np.unique(meta_u["label_int"])) assert set(np.unique(meta_u["label_int"])) == {-1} print("Unlabeled metadata columns:", meta_u.columns.tolist()) print("Unlabeled split loaded successfully.") ## Jupyter笔记本 本仓库附带了`.ipynb`格式的探索笔记本，旨在提供净化版数据集的额外背景信息、结构说明与主要分析用例。该笔记本可用于： - 查看标记与未标记划分集 - 探索元数据字段与标签分布 - 验证数据集一致性 - 查看示例分析与下游用例 若需本地打开该笔记本，请运行： bash jupyter notebook ember_cleaned_dataset.ipynb 或使用JupyterLab时运行： bash jupyter lab ember_cleaned_dataset.ipynb 请确保从数据集根目录打开笔记本，以确保相对文件路径正确解析。 ## 典型应用场景 EMBER Cleaned 支持以下任务： - 二元恶意软件检测 - 表格型机器学习流水线基准测试 - 特征重要性分析 - 基于独立未标记划分集的半监督学习 - 探索性数据分析 - 表征学习与聚类 附带的笔记本包含数据集加载、探索性分析以及针对判别特征与模型评估的示例用例。 ## 注意事项与局限性 本数据集仅支持静态分析任务。 本净化版数据集包含衍生特征，而非原始PE二进制文件。 未标记划分集不可视为真实标签。 若未经过额外验证，基于EMBER的研究结果不可泛化至现代恶意软件场景。 本数据集仅用于防御性研究、基准测试与教育用途。 ## 许可证 本净化版数据集源自EMBER。原始EMBER数据文件采用MIT许可证。请确保您的下游再分发与复用符合原始EMBER的许可证条款。 ## 引用说明 若使用本数据集，请引用原始EMBER文献： bibtex @article{anderson2018ember, title={EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models}, author={Anderson, Hyrum S. and Roth, Phil}, journal={arXiv preprint arXiv:1804.04637}, year={2018} } ## APA格式引用： Anderson, H. S., & Roth, P. (2018). EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models. arXiv. https://doi.org/10.48550/arXiv.1804.04637 ## 联系方式 共享方：ACN
