AdaMLLab/HinMix

---
language:
- hi
license: other
task_categories:
- text-generation
arxiv: 2512.18834
configs:
- config_name: minhash_deduped
data_files:
- split: train
path: minhash_deduped/**/*.parquet
- config_name: quality_filtered
data_files:
- split: train
path: quality_filtered/**/*.parquet
- config_name: matched
data_files:
- split: train
path: consensus/*.parquet
default: minhash_deduped
---

<img src="https://huggingface.co/datasets/AdaMLLab/HinMix/resolve/main/finetasks_hindi_main_results.png" width="900" alt="Finetasks benchmark scores, showing HinMix-MinHash as SOTA.">

<p align="center">
<a href="https://huggingface.co/collections/AdaMLLab/mixminmatch">
<img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="MixMinMatch Collection">
</a>
</p>

HinMix ([https://arxiv.org/abs/2512.18834](https://arxiv.org/abs/2512.18834)) is a Hindi pretraining corpus containing 76 billion tokens across 60 million documents (in the minhash subset). Rather than scraping the web again, HinMix combines six publicly available Hindi datasets, applies Hindi-specific quality filtering, and performs cross-dataset deduplication.

We train a 1.4B parameter language model through nanotron on 30 billion tokens to show that HinMix outperforms the previous state-of-the-art, [CulturaX Hindi](https://huggingface.co/datasets/uonlp/CulturaX) (see [Appendix A9 in the Fineweb-2 paper](https://arxiv.org/pdf/2506.20920)). The `minhash_deduped` subset achieves an 11.6% relative improvement, while the `matched` subset achieves an 8.1% relative improvement.

## Subsets

| Subset | Documents | Tokens | Description |
|--------|-----------|--------|-------------|
| `quality_filtered` | 99.6M | 130.3B | Quality-filtered data before deduplication |
| `minhash_deduped` | 59.6M | 76.2B | Document-level MinHash deduplication |
| `matched` | 19.8M | 27.1B | Documents appearing in 2+ source datasets |

The matched subset uses cross-dataset agreement as a signal for quality.

## Usage

```python
from datasets import load_dataset

ds = load_dataset("AdaMLLab/HinMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/HinMix", "quality_filtered")
ds = load_dataset("AdaMLLab/HinMix", "matched")
```

## Sources

Tokens were counted using `meta-llama/Llama-3.2-3B`'s tokenizer.

| Source | Tokens (MinHash) | Documents (MinHash) |
|--------|------------------|---------------------|
| FineWeb-2 | 20.0B | 17.1M |
| CulturaX | 16.6B | 11.5M |
| Sangraha (unverified) | 11.5B | 8.9M |
| HPLT 2.0 | 10.2B | 6.7M |
| Sangraha (verified) | 10.1B | 9.1M |
| C4 | 7.7B | 6.3M |
| **Total** | **76.2B** | **59.6M** |

## Pipeline

1. Quality filtering with Hindi-specific thresholds (Devanagari script ratio, repetition patterns, language identification)
2. Document-level MinHash deduplication (5-gram shingles, 14 bands, 8 hashes per band, similarity threshold 0.8)
3. Cross-source matching to identify documents appearing in 2+ independent sources

## Citation

```bib
@misc{alrashed2025mixminmatch,
title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
author={Sultan Alrashed and Francesco Orabona},
year={2025},
eprint={2512.18834v2},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.18834v2},
}
```

## License

See individual source dataset licenses.

数据集元信息：
- 语言：印地语（ISO 639-1代码：hi）
- 许可证：其他
- 任务类别：文本生成
- arXiv编号：2512.18834
- 配置项：
1. 配置名称：`minhash_deduped`
数据文件：
- 拆分：训练集
路径：`minhash_deduped/**/*.parquet`
2. 配置名称：`quality_filtered`
数据文件：
- 拆分：训练集
路径：`quality_filtered/**/*.parquet`
3. 配置名称：`matched`
数据文件：
- 拆分：训练集
路径：`consensus/*.parquet`
- 默认配置：`minhash_deduped`

<div align="center">
<a href="https://huggingface.co/collections/AdaMLLab/mixminmatch">
<img src="https://img.shields.io/badge/🤗_Collection-MixMinMatch-blue" alt="🤗 集合：MixMinMatch">
</a>
</div>

### 数据集概述
HinMix（论文链接：[https://arxiv.org/abs/2512.18834](https://arxiv.org/abs/2512.18834)）是一款印地语预训练语料库，其MinHash子集包含6000万份文档，总计762亿Token。该语料库未重新进行网络爬取，而是整合了6个公开可用的印地语数据集，通过针对印地语的专属质量过滤流程与跨数据集去重操作构建而成。

我们通过`nanotron`训练框架，在300亿Token数据上训练了一款14亿参数的大语言模型（Large Language Model, LLM），实验结果证明HinMix的性能优于此前的当前最优（State-of-the-Art, SOTA）数据集[CulturaX 印地语](https://huggingface.co/datasets/uonlp/CulturaX)（详细对比参见《Fineweb-2》论文附录A9，[https://arxiv.org/pdf/2506.20920](https://arxiv.org/pdf/2506.20920)）。其中`minhash_deduped`子集相对性能提升11.6%，`matched`子集相对性能提升8.1%。

### 数据集子集
| 子集名称 | 文档数量 | Token总量 | 子集描述 |
|---------|---------|---------|---------|
| `quality_filtered` | 9960万 | 1303亿 | 去重前完成质量过滤的原始数据集 |
| `minhash_deduped` | 5960万 | 762亿 | 经过文档级MinHash（最小哈希）去重后的数据集 |
| `matched` | 1980万 | 271亿 | 在2个及以上独立源数据集中出现的文档 |

`matched`子集以跨数据集一致性作为质量评估信号。

### 使用示例
python
from datasets import load_dataset

ds = load_dataset("AdaMLLab/HinMix", "minhash_deduped")
ds = load_dataset("AdaMLLab/HinMix", "quality_filtered")
ds = load_dataset("AdaMLLab/HinMix", "matched")

### 数据来源
本次数据集的Token计数采用`meta-llama/Llama-3.2-3B`的分词器完成，各源数据集的统计信息如下：

| 数据源 | MinHash子集Token数 | MinHash子集文档数 |
|-------|------------------|-------------------|
| FineWeb-2 | 200亿 | 1710万 |
| CulturaX | 166亿 | 1150万 |
| Sangraha（未验证版本） | 115亿 | 890万 |
| HPLT 2.0 | 102亿 | 670万 |
| Sangraha（已验证版本） | 101亿 | 910万 |
| C4 | 77亿 | 630万 |
| **总计** | **762亿** | **5960万** |

### 数据处理流程
1. **印地语专属质量过滤**：基于天城文脚本占比、重复模式、语言识别等专属阈值进行数据清洗；
2. **文档级MinHash去重**：采用5-gram分片、14个分组、每组8个哈希函数，相似度阈值设置为0.8；
3. **跨源匹配**：识别出在2个及以上独立数据源中出现的文档。

### 引用格式
bib
@misc{alrashed2025mixminmatch,
title={Mix, MinHash, and Match: Cross-Source Agreement for Multilingual Pretraining Datasets},
author={Sultan Alrashed and Francesco Orabona},
year={2025},
eprint={2512.18834v2},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2512.18834v2},
}

### 许可证
请参阅各独立源数据集的官方许可证条款。