TempestTeam/dataset-tokenizer

该数据集旨在用于训练能够对文本和代码数据进行分类的模型。它提供了一个多样化的自然语言和代码样本集合，用于自然语言处理（NLP）和代码理解等任务。数据集支持三种自然语言（英语、法语、西班牙语）和五种编程语言（Python、Java、JavaScript、C++、C）。每个数据条目包括一个唯一标识符、原始内容、语言类型和内容类型。适用于多种自然语言和编程语言的文本分类、情感分析、代码分类、代码补全等任务。

This dataset is designed for training models to classify both textual and code data. It provides a diverse collection of natural language and code samples for tasks such as natural language processing (NLP) and code understanding. The dataset supports three natural languages (English, French, Spanish) and five programming languages (Python, Java, JavaScript, C++, C). Each entry includes a unique identifier, original content, language type, and content type. It is suitable for tasks like text classification, sentiment analysis, code classification, and code completion across multiple languages.