Basanta55/cc100-nepali-strictly-cleaned-devanagari-only

CC-100尼泊尔语数据集（仅天城文清洗版）是一个从CC-100多语言语料库中专门提取和清洗的尼泊尔语文本集合。该数据集经过严格处理，仅包含天城文（Devanagari）脚本的文本，以确保语言纯度和一致性。清洗流程包括Unicode标准化（NFC和ftfy处理）、基于规则的过滤（如文本长度、天城文字符比例≥0.5、去除模板内容）、语言识别（使用fastText模型，置信度阈值≥0.7）、精确去重（基于MD5哈希）和近似去重（基于字符13-gram布隆过滤器）。最后，数据被划分为98%训练集、1%验证集和1%测试集，使用固定种子42以确保可重复性。该数据集适用于自然语言处理任务，如语言建模、文本分类和机器翻译，特别针对尼泊尔语研究。

CC-100 Nepali — Cleaned (Devanagari Only) is a meticulously processed text dataset derived from the CC-100 multilingual corpus, focusing exclusively on Nepali language content written in the Devanagari script. The dataset undergoes a comprehensive cleaning pipeline, including Unicode normalization (NFC and ftfy), rule-based filtering (e.g., text length, Devanagari character ratio ≥ 0.5, boilerplate removal), language identification (using fastText lid.176.bin with confidence ≥ 0.7), exact deduplication (via MD5 hashing), and near-deduplication (using a character 13-gram bloom filter). It is split into 98% training, 1% validation, and 1% test sets with a fixed seed of 42 for reproducibility. This dataset is suitable for NLP tasks such as language modeling, text classification, and machine translation, particularly for Nepali language research.