bot-yaya/UPRPRC_BLOCKWISE

该数据集是一个多语言文本数据集，包含阿拉伯语、中文、德语、英语、世界语、西班牙语、法语、希伯来语、印尼语、意大利语、日语、韩语、荷兰语、葡萄牙语、俄语、瑞典语、泰语、越南语和中文（MD5加密版本）等语言的文本。此外，数据集还包含了段落数、低质量段落数、去重段落数、文件名、时间戳、是否待查文件、是否跨文件重复、是否重复、是否重复文件等元数据信息。数据集分为训练集，其大小为61374874324字节，共有36700842个示例。

This dataset is a multilingual text dataset containing texts in Arabic, Chinese, German, English, Esperanto, Spanish, French, Hebrew, Indonesian, Italian, Japanese, Korean, Dutch, Portuguese, Russian, Swedish, Thai, Vietnamese, and Chinese (MD5 encrypted version). In addition, the dataset includes metadata such as the number of paragraphs, number of low-quality paragraphs, number of deduplicated paragraphs, file name, timestamp, whether it is a file to be checked, whether it is duplicated across files, whether it is duplicated, whether it is a duplicate file. The dataset is split into a training set, which is 61374874324 bytes in size and contains 36700842 examples.