VTSNLP/vietnamese_curated_dataset

越南语精选文本数据集。该数据集从多个公开的越南语数据集中收集，并使用NeMo Curator进行整理。数据集由Viettel Solutions开发，语言为越南语。数据集的收集来源包括C4数据集的越南语子集、OSCAR数据集的越南语子集、越南语维基百科文章以及Binhvq的越南语新闻语料库。预处理步骤包括Unicode重新格式化、精确去重、质量过滤、启发式过滤和基于分类器的过滤。

Vietnamese Curated Text Dataset. This dataset is collected from multiple open Vietnamese datasets, and curated with NeMo Curator. The dataset is developed by Viettel Solutions, with the language being Vietnamese. The data collection includes the Vietnamese subset of the C4 dataset, the Vietnamese subset of the OSCAR dataset, Vietnamese articles from Wikipedia, and the Vietnamese news corpus by Binhvq. The preprocessing steps include Unicode reformatting, exact deduplication, quality filtering, heuristic filtering, and classifier-based filtering. Dataset statistics include content diversity, character-based metrics, token count distribution, and embedding visualization.