TesseraeAnon/tesserae-dataset

Tesserae数据集是当前最大的公开可用的文件片段分类数据集，专为数字取证研究设计。它包含来自619种不同文件类型的原始字节片段，每个样本是从已知类型的文件中提取的512字节原始片段。该数据集旨在开发和基准测试基于内容的文件类型识别（FFTI）方法，分类任务仅基于原始字节识别文件类型，不使用元数据、文件头或文件名。数据集经过内容去重处理，采用分割感知的幸存者选择策略（优先级：训练集 > 验证集 > 测试集），确保去重后的测试分区不包含训练集或验证集中的内容，消除了跨分割内容泄漏。数据集总计包含1,878,712,731个512字节块，块数据类型为uint8，来自57,332,398个唯一源文件，类别不平衡比率为23,046,717:1，反映了实际取证场景中的文件类型分布。数据集包括源代码、标记、配置、二进制可执行文件、归档、媒体、文档等多种类别，并提供了预计算的训练/验证/测试分割（在文件级别进行分层分割以防止数据泄漏）。数据集结构包括数据文件（如block.npy、filetype_id.npy等）和分割文件（如train_indices.npy等），支持内存映射访问以进行大规模训练。所有数据均来自具有各种宽松许可证的文件，使用必须遵守原始许可证条款。The Tesserae dataset is the largest publicly available file fragment classification dataset, specifically designed for digital forensics research. It contains raw byte fragments from 619 distinct file types, with each sample being a 512-byte raw fragment extracted from a file of known type. This dataset is intended for developing and benchmarking content-based file type identification (FFTI) methods, where the classification task identifies file types solely based on raw bytes, without using metadata, file headers, or filenames. The dataset has undergone content deduplication, adopting a split-aware survivor selection strategy with the priority order: training set > validation set > test set, ensuring that the deduplicated test partition contains no content from the training or validation sets, thereby eliminating cross-partition data leakage. In total, the dataset includes 1,878,712,731 512-byte blocks with a data type of uint8, sourced from 57,332,398 unique source files. The class imbalance ratio is 23,046,717:1, which reflects the file type distribution in real-world forensics scenarios. The dataset covers multiple categories including source code, markup, configuration files, binary executables, archives, media, documents and more, and provides precomputed train/validation/test splits (stratified at the file level to prevent data leakage). The dataset structure consists of data files (e.g., block.npy, filetype_id.npy, etc.) and split files (e.g., train_indices.npy, etc.), supporting memory-mapped access for large-scale training. All data is derived from files under various permissive licenses, and users must comply with the original license terms when using the dataset.