credi-net/CleanCDB

CrediBench Cleaned（CDB Cleaned）是CrediBench数据集的预处理版本，专门用于文本预训练。该数据集包含来自网站域名的文本内容，这些内容提取自CommonCrawl的WET文件，并经过多步预处理，包括Unicode修复、换行符规范化、URL移除、模糊去重、语言识别（使用FastText模型）和质量过滤。数据集支持约143种语言，其中英语、德语和中文是最主要的语言。统计数据表明，数据集涵盖约115M个域名，包含155B个令牌，其中37M个为唯一域名。预处理步骤显著减少了文本大小，并去除了重复内容，例如CloudFlare防护消息等。此外，数据集还提供了令牌长度分布和截断率分析，以支持模型训练中的序列长度选择。CrediBench Cleaned (CDB Cleaned) is the pre-processed version of the CrediBench dataset, curated for text pre-training. It contains text content from website domains, extracted from CommonCrawl WET files, and undergoes multiple pre-processing steps including Unicode fixing, newline normalization, URL removal, fuzzy deduplication, language identification (using the FastText model), and quality filtering. The dataset supports approximately 143 languages, with English, German, and Chinese being the top three. Statistics show that the dataset covers about 115M domains, containing 155B tokens, with 37M unique domains. The pre-processing steps significantly reduce text size and remove duplicate content, such as CloudFlare protection messages. Additionally, the dataset provides token length distributions and truncation rate analyses to support sequence length selection in model training.