ccaligned_multilingual

CCAligned consists of parallel or comparable web-document pairs in 137 languages aligned with English. These web-document pairs were constructed by performing language identification on raw web-documents, and ensuring corresponding language codes were corresponding in the URLs of web documents. This pattern matching approach yielded more than 100 million aligned documents paired with English. Recognizing that each English document was often aligned to mulitple documents in different target language, we can join on English documents to obtain aligned documents that directly pair two non-English documents (e.g., Arabic-French).

CCAligned 数据集包含与英语对齐的 137 种语言的平行或可比网络文档对。这些网络文档对是通过在原始网络文档上执行语言识别并确保网络文档的 URL 中对应的语言代码相匹配而构建的。这种模式匹配方法产生了超过 1 亿份与英语对齐的文档对。鉴于每个英语文档通常与不同目标语言中的多个文档对齐，我们可以通过英语文档进行连接，以获取直接配对两种非英语文档的对齐文档（例如，阿拉伯语-法语）。