AdaMLLab/KorMix

KorMix是一个韩语预训练语料库，通过结合五个公开可用的韩语数据集构建而成，应用了韩语特定的质量过滤，并执行了跨数据集去重。数据集包含三个子集：quality_filtered（去重前的质量过滤数据）、minhash_deduped（文档级MinHash去重数据）、matched（出现在2个以上源数据集中的文档，使用跨数据集一致性作为质量信号）。构建流程包括：使用韩语特定阈值（如韩文脚本比例、无空格语言调整用于词级指标、重复模式）进行质量过滤；文档级MinHash去重（使用5-gram shingles、14个band、每个band 8个哈希、相似度阈值0.8）；跨源匹配以识别在多个独立来源中出现的文档。数据来源包括：FineWeb-2、HPLT 2.0、CulturaX、C4和FinePDFs。KorMix is a Korean pre-training corpus constructed by integrating five publicly available Korean datasets, with Korean-specific quality filtering applied and cross-dataset deduplication implemented. The dataset includes three subsets: quality_filtered (quality-filtered data prior to deduplication), minhash_deduped (document-level MinHash deduplicated data), and matched (documents appearing in more than two source datasets, where cross-dataset consistency is used as a quality signal). The construction pipeline consists of the following steps: quality filtering using Korean-specific thresholds such as Korean script proportion, whitespace-free language adjustment for word-level metrics and duplicate patterns; document-level MinHash deduplication using 5-gram shingles, 14 bands, 8 hashes per band and a similarity threshold of 0.8; and cross-source matching to identify documents present across multiple independent sources. The data sources include FineWeb-2, HPLT 2.0, CulturaX, C4 and FinePDFs.