nrl-ai/vn-diacritic-train

这是一个用于越南语重音符号恢复任务的训练数据集，包含两个不同的配置：wiki_500k和news_150k。wiki_500k包含50万对（输入，目标）文本对，来自维基百科语料库，覆盖广泛的百科主题。news_150k包含15万对文本对，来自现代新闻/商业语料库，补充了wiki_500k的百科性质。每个数据条目都是一个JSONL记录，包含去除了重音符号的输入文本和带有正确重音符号的目标文本。数据集经过了严格的过滤和处理，包括句子长度筛选、ASCII比例筛选、去重等步骤，以确保质量和多样性。

This is a training dataset for Vietnamese diacritic-restoration tasks, containing two distinct configurations: wiki_500k and news_150k. wiki_500k includes 500K (input, target) pairs from a Wikipedia corpus, covering a broad range of encyclopedic topics. news_150k includes 150K pairs from a modern news/business corpus, complementing the encyclopedic nature of wiki_500k. Each data entry is a JSONL record with diacritic-stripped input text and the original correctly diacriticized target. The dataset has undergone rigorous filtering and processing, including sentence length filtering, ASCII ratio filtering, deduplication, etc., to ensure quality and diversity.