ashtok897/indic-hplt-v1

Indic HPLT v1 是一个多语言预训练语料库，包含 9,836,075 个文档（约 84 亿个估计词元），涵盖 10 种印度语言（印地语、孟加拉语、泰米尔语、泰卢固语、马拉地语、古吉拉特语、卡纳达语、马拉雅拉姆语、旁遮普语、乌尔都语）和英语。该数据集基于 HPLT Monolingual v3 高质量网络爬取数据构建，适用于文本生成和填充掩码等自然语言处理任务。数据经过质量过滤（如字符数限制、非字母字符比例控制、平均词长要求）和去重处理（包括精确哈希去重和 MinHash LSH 去重），但仅包含网络文本，未包含书籍、维基百科或结构化数据，且包含约 8.3% 的机器翻译内容。数据分为训练集（约 960 万文档）、验证集（约 10 万文档）和测试集（约 3.6 万文档），每个文档包含文本、语言代码、源 URL、质量分数等字段。

A multilingual pretraining corpus of 9,836,075 documents (~8.4B estimated tokens) across 10 Indic languages and English, built from HPLT Monolingual v3 high-quality web crawl data. The dataset is designed for text generation and fill-mask tasks, with quality filtering (e.g., character limits, non-alphabetic character ratio, average word length) and deduplication (including exact hash deduplication and MinHash LSH for English). It contains only web text, no books, Wikipedia, or structured data, and includes approximately 8.3% machine-translated content. The data is split into train (~9.6M documents), validation (~100K documents), and test (~36K documents) sets, with fields such as text, language code, source URL, quality score, and more.