agentlans/multilingual-text

这个数据集包含多种语言的文本数据，每个配置都对应一个特定的语言和一个训练分割文件。数据集经过预处理，确保了高质量的语言块。预处理步骤包括归一化空白字符、标点符号、Unicode字符和项目符号，以及替换敏感信息。使用FastText模型进行语言检测，并移除重复块。

This dataset contains text data in multiple languages, with each configuration corresponding to a specific language and a training split file. The dataset has undergone preprocessing to ensure high-quality language chunks. Preprocessing steps include normalization of whitespace, punctuation, Unicode characters, and bullet points, as well as the replacement of sensitive information. Language detection is performed using the FastText model, and duplicate chunks are removed.