catallama/Catalan-Instruct-V2

Catalan Instruct V2数据集是Catalan Instruct数据集的第二版，比第一版大2.5倍，并包含从多个公共数据集中筛选的高质量指令。此外，还包含了25万条从加泰罗尼亚语翻译成英语的新指令，使得加泰罗尼亚语指令总数超过46万条。数据集总共包含709,000个样本指令，使用Llama-3 Tokenizer进行标记化后总计289M个标记。数据集的任务包括信息提取、命名实体识别、翻译、摘要、聊天、情感分析和开放问答。数据来源包括多个公开数据集，如projecte-aina/InstruCAT、projecte-aina/RAG_Multilingual等。数据集中加泰罗尼亚语占55%，英语占45%。

The Catalan Instruct V2 Dataset is a new iteration on the Catalan Instruct dataset. The V2 is 2.5x larger than V1 and it contains high quality instructions filtered from multiple public datasets. It also contains an additional 250k new translated instructions from Catalan to English, taking the total number of Catalan instructions to 460k+. In total, the dataset contains 709k sample instructions totalling 289M tokens after tokenizing it with the Llama-3 Tokenizer. The dataset is an expansion of the V1, where all the English instruction were replaced with high quality, filtered instructions and an additional 250k Catalan Instructions were translated and added. The tasks include information extraction, named entity recognition, translation, summarization, chat, sentiment analysis, and open question answering. Data sources include multiple public datasets such as projecte-aina/InstruCAT, projecte-aina/RAG_Multilingual, etc. The dataset is 55% Catalan and 45% English.

原始信息汇总

数据集概述

名称: Catalan Instruct V2

版本: V2

大小:

• 总样本数: 709k
• 总token数: 289M
• 数据集大小: 1067643929 bytes
• 下载大小: 591460059 bytes

语言:

• 加泰罗尼亚语 (ca-ES) - 55%
• 英语 (en-US) - 45%

特征:

• messages: 包含 content (字符串) 和 role (字符串)
• category: 字符串
• language: 字符串

数据分割:

• train: 695757 样本, 1046289757.139169 bytes
• test: 14200 样本, 21354171.860831007 bytes

任务类别:

• 文本生成
• 信息提取 (适用于RAG)
• 命名实体识别 (NER)
• 英加泰罗尼亚语互译
• 摘要生成 (短形式和长形式)
• 聊天
• 情感分析
• 开放式问答

许可证: Creative Commons Attribution 4.0 International (cc-by-sa-4.0)