AdaMLLab/ThaMix

---
language:
- th
license: other
task_categories:
- text-generation
arxiv: 2512.18834
configs:
- config_name: minhash_deduped
data_files:
- split: train
path: minhash_deduped/**/*.parquet
- config_name: quality_filtered
data_files:
- split: train
path: quality_filtered/**/*.parquet
- config_name: matched
data_files:
- split: train
path: consensus/*.parquet
default: minhash_deduped
---

<p align="center">
<a href="https://huggingface.co/collections/AdaMLLab/mixminmatch">
<img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="MixMinMatch Collection">
</a>
</p>

ThaMix ([https://arxiv.org/abs/2512.18834](https://arxiv.org/abs/2512.18834)) is a Thai pretraining corpus built by combining seven publicly available Thai datasets, applying Thai-specific quality filtering, and performing cross-dataset deduplication.

## Subsets

| Subset | Description |
|--------|-------------|
| `quality_filtered` | Quality-filtered data before deduplication |
| `minhash_deduped` | Document-level MinHash deduplication |
| `matched` | Documents appearing in 2+ source datasets |

The matched subset uses cross-dataset agreement as a signal for quality.

## Usage

```python
from datasets import load_dataset

ds = load_dataset("AdaMLLab/ThaMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/ThaMix", "quality_filtered")
ds = load_dataset("AdaMLLab/ThaMix", "matched")
```

## Sources

- FineWeb-2 (`HuggingFaceFW/fineweb-2`, `tha_Thai`)
- HPLT 2.0 (`HPLT/HPLT2.0_cleaned`, `tha_Thai`)
- CulturaX (`uonlp/CulturaX`, `th`)
- C4 (`allenai/c4`, `th`)
- SEA CommonCrawl (`sailor2/sea-commoncrawl`, `thai`)
- FinePDFs (`HuggingFaceFW/finepdfs`, `tha_Thai`)
- SEA-LION Pile v2 (`aisingapore/SEA-PILE-v2`, `th`)

## Pipeline

1. Quality filtering with Thai-specific thresholds (Thai script ratio, repetition patterns, line quality)
2. Document-level MinHash deduplication (5-gram shingles, 14 bands, 8 hashes per band, similarity threshold 0.8)
3. Cross-source matching to identify documents appearing in 2+ independent sources

## Citation

```bib
@misc{alrashed2025mixminmatch,
title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
author={Sultan Alrashed and Francesco Orabona},
year={2025},
eprint={2512.18834v2},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.18834v2},
}
```

## License

See individual source dataset licenses.

language:
- 泰语（th）
license: 其他
task_categories:
- 文本生成
arxiv: 2512.18834
configs:
- config_name: minhash_deduped
data_files:
- split: 训练集
path: minhash_deduped/**/*.parquet
- config_name: quality_filtered
data_files:
- split: 训练集
path: quality_filtered/**/*.parquet
- config_name: matched
data_files:
- split: 训练集
path: consensus/*.parquet
default: minhash_deduped

<p align="center">
<a href="https://huggingface.co/collections/AdaMLLab/mixminmatch">
<img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="🤗 集合-MixMinMatch">
</a>
</p>

ThaMix（[https://arxiv.org/abs/2512.18834](https://arxiv.org/abs/2512.18834)）是一款泰语预训练语料库，通过整合七个公开可用的泰语数据集、应用针对泰语定制的质量过滤策略，并执行跨数据集去重操作构建而成。

## 子集

| 子集名称 | 描述 |
|--------|-------------|
| `quality_filtered` | 去重前经过质量过滤的数据集 |
| `minhash_deduped` | 经过文档级最小哈希（MinHash）去重的数据集 |
| `matched` | 在2个及以上源数据集中出现的文档 |

matched子集以跨数据集一致性作为质量评估信号。

## 使用方法

python
from datasets import load_dataset

ds = load_dataset("AdaMLLab/ThaMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/ThaMix", "quality_filtered")
ds = load_dataset("AdaMLLab/ThaMix", "matched")

## 数据源

- FineWeb-2 (`HuggingFaceFW/fineweb-2`, `tha_Thai`)
- HPLT 2.0 (`HPLT/HPLT2.0_cleaned`, `tha_Thai`)
- CulturaX (`uonlp/CulturaX`, `th`)
- C4 (`allenai/c4`, `th`)
- SEA CommonCrawl (`sailor2/sea-commoncrawl`, `thai`)
- FinePDFs (`HuggingFaceFW/finepdfs`, `tha_Thai`)
- SEA-LION Pile v2 (`aisingapore/SEA-PILE-v2`, `th`)

## 处理流程

1. 基于针对泰语的定制阈值开展质量过滤（涵盖泰语字符占比、重复模式、文本行质量）
2. 文档级最小哈希（MinHash）去重（采用5-gram分片、14个分桶、每个分桶8个哈希值，相似度阈值设为0.8）
3. 执行跨源匹配，以识别出现在2个及以上独立数据源中的文档

## 引用格式

bib
@misc{alrashed2025mixminmatch,
title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
author={Sultan Alrashed and Francesco Orabona},
year={2025},
eprint={2512.18834v2},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.18834v2},
}

## 许可证

请参阅各源数据集的许可证条款。