Research-EAI/essential-web-1t-sample-fdc-partitioned

本数据集包含从Essential-Web数据集中提取的1万亿个token样本，并按照Free Decimal Correspondence (FDC)二级分类进行划分。Essential-Web是一个包含24万亿个token的网页数据集，具有广泛的文档级元数据，旨在通过类似SQL的筛选方式实现快速数据集整理。FDC分类法是一种开放分类系统，灵感来自杜威十进制分类法。二级分类提供了广泛的主题分类，使研究人员能够快速识别和筛选相关的内容领域。数据集的创建使用了在合成标签上训练的分类器，并涉及23.6亿个网页文档的推理，需要大约90,000 AMD MI300x GPU小时。使用简单元数据过滤器从Essential-Web数据集中整理的数据集在与顶级网页整理数据集相比时表现出竞争性性能。数据集按FDC二级分类组织，每个分区包含带有相应FDC分类和关联分类法元数据的文档。

This dataset contains a 1 trillion token sample from Essential-Web, partitioned by Free Decimal Correspondence (FDC) level-2 categories. Essential-Web is a 24-trillion-token web dataset with extensive document-level metadata designed to enable rapid dataset curation through SQL-like filtering. The FDC taxonomy is an open classification system inspired by the Dewey Decimal System, providing broad subject matter classifications for quick identification and filtering of relevant content domains. The dataset was created using a classifier trained on synthetic labels, with performance metrics demonstrating competitive performance relative to top web-curated datasets. The dataset structure is organized by FDC level-2 categories, and the schema documentation details fields and classification systems used, including EAI Taxonomy, Blooms Taxonomy, and content quality dimensions. Metadata fields provide information about the source document, such as URL, domain, and archive details.