abir-hr196/multilingual_data_tokenized

这是一个包含多种语言文本的数据集，具体包括德语（deu_Latn_tokenized）、阿拉伯语（arb_Arab_tokenized）、法语（fra_Latn_tokenized）、中文（cmn_Hani_tokenized）和英语（eng_Latn_tokenized）。每个语言都有相应的文本、语言类型、input_ids和attention_mask等信息。数据集分为不同的部分，每个部分包含特定数量的示例和字节数。

This dataset contains text in multiple languages, including German (deu_Latn_tokenized), Arabic (arb_Arab_tokenized), French (fra_Latn_tokenized), Chinese (cmn_Hani_tokenized), and English (eng_Latn_tokenized). Each language has corresponding text, language type, input_ids, and attention_mask information. The dataset is split into different sections, each containing a specific number of examples and bytes.