CCAligned-multilingual
收藏OpenXLab2026-04-18 收录
下载链接：
https://openxlab.org.cn/datasets/OpenDataLab/ccaligned-multilingual
下载链接
链接失效反馈官方服务：
资源简介：
CCAligned consists of parallel or comparable web-document pairs in 137 languages aligned with English. These web-document pairs were constructed by performing language identification on raw web-documents, and ensuring corresponding language codes were corresponding in the URLs of web documents. This pattern matching approach yielded more than 100 million aligned documents paired with English. Recognizing that each English document was often aligned to mulitple documents in different target language, we can join on English documents to obtain aligned documents that directly pair two non-English documents (e.g., Arabic-French). This corpus was created from 68 Commoncrawl Snapshots.
To load a language which isn't part of the config, all you need to do is specify the language code. You can find the valid languages in http://www.statmt.org/cc-aligned/ E.g.
dataset = load-dataset("ccaligned-multilingual", language-code="fr-XX", type="documents")
提供机构：
OpenDataLab
创建时间：
2023-12-07
