kaihe/chinese_vietnamese_bilingual_wangwen

本数据集是一个中文到越南语的机器翻译数据集，来源于网络小说。通过对比中文和越南语版本的小说内容进行对齐，包括章节对齐和句子对齐。对齐过程使用了基于实体识别和相似度计算的算法。数据集包含了90本小说的双语对齐数据，分为篇章对齐和句子对齐两种格式。

This dataset is a Chinese to Vietnamese machine translation dataset, primarily sourced from web novels. The construction process of the dataset involves selecting popular novels from Vietnamese novel sites and finding corresponding Chinese versions for alignment. The alignment process includes chapter alignment and sentence alignment, using an alignment algorithm designed by the author, referencing transportation theory. Similarity calculation is key, with chapter-level similarity based on cosine distance of entity word vectors, and sentence-level similarity based on bilingual dictionaries or Siamese networks. The dataset structure includes JSON files for chapter alignment and sentence alignment, as well as a ZIP file containing 90 bilingual web novels and Vietnamese entity words.