AdaMLLab/JpnMix

JpnMix是一个日语预训练语料库，通过结合五个公开可用的日语数据集构建而成，应用了日语特定的质量过滤，并执行了跨数据集去重。该数据集包括三个子集：quality_filtered（去重前的质量过滤数据）、minhash_deduped（文档级MinHash去重数据）和matched（出现在两个或更多源数据集中的文档，使用跨数据集一致性作为质量信号）。数据来源包括FineWeb-2、HPLT 2.0、CulturaX、C4和FinePDFs。处理流程包括：使用日语特定阈值进行质量过滤（如CJK+假名比例、无空格语言调整以用于词级指标、重复模式），文档级MinHash去重（使用5-gram shingles、14个band、每个band 8个哈希、相似度阈值0.8），以及跨源匹配以识别出现在两个或更多独立源中的文档。JpnMix is a Japanese pretraining corpus built by combining five publicly available Japanese datasets, applying Japanese-specific quality filtering, and performing cross-dataset deduplication. It includes subsets: quality_filtered (quality-filtered data before deduplication), minhash_deduped (document-level MinHash deduplication), and matched (documents appearing in 2+ source datasets, using cross-dataset agreement as a signal for quality). Sources include FineWeb-2, HPLT 2.0, CulturaX, C4, and FinePDFs. The pipeline involves quality filtering with Japanese-specific thresholds (CJK+Kana script ratio, no-space-language adjustments for word-level metrics, repetition patterns), document-level MinHash deduplication (5-gram shingles, 14 bands, 8 hashes per band, similarity threshold 0.8), and cross-source matching to identify documents appearing in 2+ independent sources.