OPUS-100

"继[3]之后，OPUS-100是以英语为中心的，这意味着所有训练对在源端或目标端都包括英语。语料库涵盖100种语言（包括英语）。我们根据体积选择语言OPUS 中可用的并行数据。OPUS 集合由多个语料库组成，从电影字幕到 GNOME 文档再到圣经。我们没有整理数据或尝试平衡不同领域的表示，而是选择了最简单的方法下载每个语言对的所有语料库并将它们连接起来。数据集分为训练、开发和测试部分。我们随机抽取每个语言对最多 100 万个句子对进行训练，每个语言对最多抽取 2000 个句子对用于开发和测试。为了确保训练和开发/测试数据之间没有重叠（在单语句子级别），我们在采样期间应用过滤器以排除已经采样的句子。注意这是跨语言完成的，例如，训练数据的葡萄牙语-英语部分中的英语句子不会出现在印地语-英语测试集中。 OPUS-100 包含大约 55M 句对。在 99 个语言对中，44 个有 1M 的句子对训练数据，73 个至少有 100k，95 个至少有 10k。"

Following [3], OPUS-100 is English-centric, meaning that all training pairs include English on either the source or target side. The corpus covers 100 languages (including English). We selected languages with available parallel data in OPUS based on corpus volume. The OPUS collection consists of multiple corpora, ranging from movie subtitles and GNOME documents to the Bible. We did not curate the data or attempt to balance the representation across different domains; instead, we adopted the simplest approach: downloading all corpora for each language pair and concatenating them. The dataset is split into training, development, and test subsets. We randomly sampled up to 1 million sentence pairs per language pair for training, and up to 2,000 per language pair for development and testing. To ensure no overlap between the training and development/test data at the monolingual sentence level, we applied filters during sampling to exclude already sampled sentences. Note that this filtering was performed across languages: for example, English sentences in the Portuguese-English subset of the training data will not appear in the Hindi-English test set. OPUS-100 contains approximately 55 million sentence pairs. Out of the 99 language pairs, 44 have 1 million sentence pairs of training data, 73 have at least 100,000, and 95 have at least 10,000.