ashtok897/indic-hplt-v2

---
language:
- hi
- bn
- ta
- te
- mr
- gu
- kn
- ml
- pa
- ur
- ne
- or
- as
- en
license: cc0-1.0
task_categories:
- text-generation
- fill-mask
pretty_name: "Indic HPLT v2 — 34.6M docs, 14 languages"
size_categories:
- 10M<n<100M
source_datasets:
- hplt-project/hplt_monolingual_v3
tags:
- indic
- multilingual
- pretraining
- hplt
---

# Indic HPLT v2

A multilingual pretraining corpus of **34,605,630 documents** (~25.5B estimated tokens, ~218 GB raw JSONL) across **13 Indic languages and English**, built from [HPLT Monolingual v3](https://hplt-project.org/datasets/v3) high-quality web crawl data.

This is the **larger successor to [Indic HPLT v1](https://huggingface.co/datasets/AM0908/indic-hplt-v1)** (9.8M docs, 11 languages). Compared to v1, this release adds 3 new Indic languages (Nepali, Odia, Assamese) and ~3.5× more documents overall.

## Quick Start

```python
from datasets import load_dataset

# Full training split
ds = load_dataset("AM0908/indic-hplt-v2", split="train")

# Filter by language
hi_ds = ds.filter(lambda x: x["lang"] == "hi")

# Streaming (recommended — corpus is ~218 GB)
ds = load_dataset("AM0908/indic-hplt-v2", split="train", streaming=True)
for row in ds.take(3):
print(row["lang"], row["text"][:100])
```

## Language Distribution

| Language | BCP-47 | Documents | Est. Tokens | Avg Words/Doc |
|---|---|---|---:|---:|
| Hindi | `hi` | 4,081,640 | ~2.43B | 464 |
| Bengali | `bn` | 3,061,224 | ~1.85B | 374 |
| Telugu | `te` | 3,061,224 | ~1.63B | 275 |
| Marathi | `mr` | 3,061,223 | ~1.79B | 348 |
| Tamil | `ta` | 3,061,220 | ~1.94B | 297 |
| Urdu | `ur` | 3,061,219 | ~1.87B | 512 |
| Kannada | `kn` | 2,806,122 | ~1.49B | 266 |
| Malayalam | `ml` | 2,806,121 | ~1.36B | 209 |
| Gujarati | `gu` | 2,806,120 | ~1.57B | 380 |
| Nepali | `ne` | 1,785,713 | ~0.85B | 296 |
| English | `en` | 1,752,198 | ~6.84B | 2,572 |
| Punjabi | `pa` | 1,517,712 | ~1.04B | 530 |
| Odia | `or` | 1,297,839 | ~0.55B | 256 |
| Assamese | `as` | 446,055 | ~0.28B | 385 |
| **Total** | | **34,605,630** | **~25.47B** | |

> Token estimates use chars÷4. Actual count varies by tokenizer; South Indian scripts (Tamil, Telugu, Kannada, Malayalam) tend to have higher tokenizer fertility, so true token counts are typically 1.3–1.8× the estimate for those languages.

## Comparison with v1

| Metric | v1 | v2 | Δ |
|---|---:|---:|---|
| Documents | 9.8M | 34.6M | **3.5×** |
| Languages | 11 | 14 | +Nepali, Odia, Assamese |
| Est. tokens | ~8.4B | ~25.5B | **3.0×** |
| English docs | 736K | 1.75M | 2.4× |
| Indic docs | 9.1M | 32.85M | 3.6× |

v1 remains useful for smaller-scale experiments or when working under tight compute budgets. v2 is intended for full pretraining runs.

## Dataset Fields

| Field | Type | Description |
|---|---|---|
| `text` | string | Document text |
| `lang` | string | BCP-47 language code |
| `url` | string | Source URL |
| `score` | float | HPLT WDS quality score (raw integer, higher = better; all docs here ≥ 10) |
| `collection` | string | Source MIME type (e.g. `text/html`) |
| `web-register` | string | Document register/genre code (see Web Registers section) |
| `prob` | float | Language detection confidence |
| `char_count` | int | Character count |
| `word_count` | int | Whitespace-split word count |
| `doc_id` | string | Unique ID e.g. `hi_0000001` |

## Data Splits

Split ratios are 98% / 1% / 1% over Parquet shards (100,000 rows per shard).

| Split | Documents | Shards |
|---|---:|---:|
| train | ~33.9M | ~339 |
| validation | ~350K | ~3 |
| test | ~350K | ~4 |

## How It Was Built

**Source:** HPLT v3 sorted shards (`https://data.hplt-project.org/three/sorted`), which order documents by WDS quality score descending — we read from the top of the sorted shards, so every document in this corpus has a WDS score of 10 or higher.

**Quality filtering** (applied inline during download):
- 50–100,000 characters per document
- Max 50% non-alphabetic characters (Unicode-aware)
- Min average word length 2.0 characters

**Deduplication:**
- Exact SHA-256 hash dedup on all languages
- MinHash near-duplicate removal (Jaccard ≥ 0.7, 128 permutations, 5-gram shingles) on English only — HPLT v3 already applies global near-deduplication to the Indic languages, so re-running MinHash on them would be wasted compute. English MinHash uses a disk-based banding implementation to keep memory flat at corpus scale.
- English: 1.75M kept after dedup (from 2.81M cleaned; ~254K near-duplicates removed)

**Merge:** Languages are interleaved according to configured fractions. Hindi is over-represented relative to its corpus share (~11.8% vs. a target near 8%) because several low-resource languages we attempted to include had insufficient data and their quota could not be redistributed within their fraction band — see Excluded Languages below.

**Pipeline code:** [github.com/ashtok/build-multilingual-corpus](https://github.com/ashtok/build-multilingual-corpus)

## Excluded Languages

These languages were targeted but **excluded** from the merge because HPLT v3 did not contain enough data to meet even 10% of the per-language quota after quality filtering:

| Language | BCP-47 | Available docs | Target | Coverage |
|---|---|---:|---:|---:|
| Maithili | `mai` | 28,641 | 2,040,816 | 1.4% |
| Bhojpuri | `bho` | 32,772 | 2,040,816 | 1.6% |
| Chhattisgarhi | `hne` | 6,317 | 1,785,714 | 0.4% |
| Manipuri (Meitei) | `mni` | 7,566 | 1,530,612 | 0.5% |
| Sanskrit | `san` | 59,152 | 1,530,612 | 3.9% |
| Santali | `sat` | 4,719 | 1,275,510 | 0.4% |

These languages exist in HPLT v3 but with too few high-quality documents to be useful in a balanced pretraining corpus. Users needing coverage for these languages should look at AI4Bharat's IndicCorp v2 or Sangraha as supplementary sources.

## Web Register Distribution

HPLT v3 labels documents with a register/genre code. Top labels in this corpus:

| Register | Count | % | Meaning |
|---|---:|---:|---|
| NA | 22.05M | 63.7% | Narrative |
| IN | 3.71M | 10.7% | Informational description |
| MT | 2.36M | 6.8% | Machine-translated |
| OP | 2.12M | 6.1% | Opinion |
| IP | 1.28M | 3.7% | Informational persuasion |
| HI | 0.64M | 1.8% | How-to / instructional |
| LY | 0.28M | 0.8% | Lyrical |
| ID | 0.25M | 0.7% | Interactive discussion |

If you want to exclude machine-translated content, filter on `web-register != "MT"` — that drops ~6.8% of the corpus.

## Limitations

- **Web text only** — no books, Wikipedia, or structured data
- **~6.8% machine-translated content** — identifiable via `web-register: MT`
- **English is still underrepresented** in document count (5.1%), though its token share is much higher (~27%) because English documents are on average ~5× longer than Indic ones in this corpus
- **Assamese is data-limited** at 446K docs — included because it meets the 10% threshold but smaller than other Indic languages
- **6 Indic languages excluded** due to insufficient HPLT v3 coverage — see Excluded Languages
- **No PII filtering beyond HPLT defaults**

## Citation

```bibtex
@dataset{indicHPLTv2_2026,
author = {Mahajan, Ashutosh},
title = {Indic {HPLT} v2: A 34.6M-document Multilingual Corpus for 13 Indic Languages and English},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/datasets/AM0908/indic-hplt-v2}
}

@article{oepen2025hplt,
title = {{HPLT} 3.0: Very Large-Scale Multilingual Resources for {LLM} and {MT}},
author = {Oepen, Stephan and others},
journal = {arXiv preprint arXiv:2511.01066},
year = {2025}
}
```

## License

[CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/) — inherited from HPLT v3.