cryscan/multilingual-share

---
license: cc0-1.0
language:
- en
- zh
---
# Multilingual Share GPT
Multilingual Share GPT, the free multi-language corpus for LLM training. All text are converted to markdown format, and classified by languages.

## Github Repo
Follow the [link](https://github.com/RWKV-Wiki/MultilingualShareGPT) here to Github.

## Data Example
```json
{
"id": "ImiMfCY",
"lang": "en",
"text": "\n<|user|>: Let's play chess\n\n<|bot|>: Sure, I'd love to play chess with you! Do you want to play a virtual game or just ask chess-related questions?\n\n<|user|>: Virtual game\n\n<|bot|>: I'm sorry, I'm an AI language model and I don't have the ability to display a chess board or move pieces on it. However, I'd be happy to help you with any chess-related questions or problems you might have. Just let me know what you need help with!\n\n<|user|>: Let's play\n\n<|bot|>: I apologize for any confusion, but as an AI language model I do not have the ability to play chess as it requires a graphical interface. However, I can help you learn and improve your chess skills by answering questions and providing explanations and analysis of specific positions or games.\n\n<|user|>: 1.e4\n\n<|bot|>: The move 1.e4 is the most common first move in the game of chess, and is known as the \"King's Pawn Opening\". This move aims to control the center of the board and create space for the other pieces to develop. Black has a variety of responses to 1.e4, but some of the most common are 1...e5 (the \"King's Gambit\"), 1...c5 (the \"Sicilian Defense\"), and 1...e6 (the \"French Defense\").\n\n<|user|>: I play 1.e4\n\n<|bot|>: Very good! You have started the game with the King's Pawn Opening. As black, I will respond with 1...e5, the King's Gambit.\n\n<|user|>: I play 2.Nf3\n\n<|bot|>: That's a good move. With 2.Nf3, you are developing your knight and attacking the black pawn on e5. As black, I'll play 2...Nc6, which develops the knight to a strong central square and prepares to castle kingside.\n\n<|user|>: I play 3.d4\n\n<|bot|>: With the move 3.d4, you are now offering a pawn to black with the King's Gambit. This gambit can lead to a sharp and tactical game, but it also weakens the white pawn structure and can lead to a disadvantage if not handled properly. As black, I will accept the gambit with 3...exd4.\n\n<|user|>: I play 4.Nxd4\n\n<|bot|>: With 4.Nxd4, you are now attacking the black pawn on d4 and forcing it to capture. As black, I'll capture the pawn with 4...Nxd4. This has now opened up the d-file for my rook and given me a lead in development.\n\n<|user|>: I play 5.Qxd4\n\n<|bot|>: With the move 5.Qxd4, you have taken control of the d-file and are attacking the black knight on d4. As black, I will respond with 5...Nc6, developing my knight to a safer square and putting pressure on the white queen.\n\n<|user|>: I play 6.Qd3\n\n<|bot|>: With 6.Qd3, you are attacking the black knight again and preparing to castle kingside. This move also has the idea of putting pressure on the d7-pawn, as well as the d8-h4 diagonal. As black, I will respond with 6...Nf6, developing my knight to a strong central square and preparing to castle kingside.\n\n<|user|>: I play 7.Nc3\n\n<|bot|>: With 7.Nc3, you are developing your knight and preparing to castle kingside. This move also has the idea of supporting the d4-queen and putting pressure on the e5-pawn. As black, I will play 7...d6, solidifying my pawn structure and preparing to develop my dark-squared bishop.\n\n<|user|>: I play 8.Bg5\n\n<|bot|>: With 8.Bg5, you are attacking the black knight on f6 and preparing to castle kingside. This move also has the idea of putting pressure on the h7-pawn and controlling the f4-square. As black, I will respond with 8...Be7, developing my dark-squared bishop and preparing to castle kingside.\n\n<|user|>: I play 9.0-0-0\n\n<|bot|>: With 9.0-0-0, you are now castling kingside and putting your king in a safer position. This move also has the idea of bringing the rook into the game more quickly. As black, I will castle kingside as well with 9...0-0-0.\n\n<|user|>: I play 10.Kb1\n\n<|bot|>: I'm sorry, but that is an illegal move. In chess, the king can only move one square at a time and cannot \"jump\" over other pieces. The correct move would be 10.Be2, developing your bishop and preparing to castle kingside.\n\n<|endoftext|>"
}
```

## 中文用户请看这里
为了推进中文AI的发展，促进AI技术公开化、国际化，我们成立了 ShareGPT-90k 项目，希望借助大家的力量推进数据清洗与对齐工作。

可能与各位想象的有所不同，GPT模型主要通过预训练数据集赋能，语料的质量对模型最终性能至关重要。然而，百度知道、CSDN、知乎等平台软文过多；小木虫等高质量平台语料过少；个人博客内容质量参差不齐。

OpenAI完成数据集的收集花费了巨大成本，以至于需要从微软集资。我们无力承担如此巨大的开销，于是需要各位有志于筹建开放获取语料，并有一定外语基础的网友们献上自己的力量。

如果您有意向参与此项目，我们将不胜感激。

项目地址：https://paratranz.cn/projects/6725

---
许可证：CC0 1.0协议
语言：
- 英语
- 中文
---
# 多语言Share GPT（Multilingual Share GPT）
多语言Share GPT是一款面向大语言模型（Large Language Model, LLM）训练的免费多语言语料库。所有文本均已转换为Markdown格式，并按语言分类。

## GitHub 仓库
点击[此链接](https://github.com/RWKV-Wiki/MultilingualShareGPT)访问其GitHub仓库。

## 数据示例
json
{
"id": "ImiMfCY",
"lang": "en",
"text": "
<|user|>: Let's play chess

<|bot|>: Sure, I'd love to play chess with you! Do you want to play a virtual game or just ask chess-related questions?

<|user|>: Virtual game

<|bot|>: I'm sorry, I'm an AI language model and I don't have the ability to display a chess board or move pieces on it. However, I'd be happy to help you with any chess-related questions or problems you might have. Just let me know what you need help with!

<|user|>: Let's play

<|bot|>: I apologize for any confusion, but as an AI language model I do not have the ability to play chess as it requires a graphical interface. However, I can help you learn and improve your chess skills by answering questions and providing explanations and analysis of specific positions or games.

<|user|>: 1.e4

<|bot|>: The move 1.e4 is the most common first move in the game of chess, and is known as "King's Pawn Opening". This move aims to control the center of the board and create space for the other pieces to develop. Black has a variety of responses to 1.e4, but some of the most common are 1...e5 (the "King's Gambit"), 1...c5 (the "Sicilian Defense"), and 1...e6 (the "French Defense").

<|user|>: I play 1.e4

<|bot|>: Very good! You have started the game with the King's Pawn Opening. As black, I will respond with 1...e5, the King's Gambit.

<|user|>: I play 2.Nf3

<|bot|>: That's a good move. With 2.Nf3, you are developing your knight and attacking the black pawn on e5. As black, I'll play 2...Nc6, which develops the knight to a strong central square and prepares to castle kingside.

<|user|>: I play 3.d4

<|bot|>: With the move 3.d4, you are now offering a pawn to black with the King's Gambit. This gambit can lead to a sharp and tactical game, but it also weakens the white pawn structure and can lead to a disadvantage if not handled properly. As black, I will accept the gambit with 3...exd4.

<|user|>: I play 4.Nxd4

<|bot|>: With 4.Nxd4, you are now attacking the black pawn on d4 and forcing it to capture. As black, I'll capture the pawn with 4...Nxd4. This has now opened up the d-file for my rook and given me a lead in development.

<|user|>: 5.Qxd4

<|bot|>: With the move 5.Qxd4, you have taken control of the d-file and are attacking the black knight on d4. As black, I will respond with 5...Nc6, developing my knight to a safer square and putting pressure on the white queen.

<|user|>: 6.Qd3

<|bot|>: With 6.Qd3, you are attacking the black knight again and preparing to castle kingside. This move also has the idea of putting pressure on the d7-pawn, as well as the d8-h4 diagonal. As black, I will respond with 6...Nf6, developing my knight to a strong central square and preparing to castle kingside.

<|user|>: 7.Nc3

<|bot|>: With 7.Nc3, you are developing your knight and preparing to castle kingside. This move also has the idea of supporting the d4-queen and putting pressure on the e5-pawn. As black, I will play 7...d6, solidifying my pawn structure and preparing to develop my dark-squared bishop.

<|user|>: 8.Bg5

<|bot|>: With 8.Bg5, you are attacking the black knight on f6 and preparing to castle kingside. This move also has the idea of putting pressure on the h7-pawn and controlling the f4-square. As black, I will respond with 8...Be7, developing my dark-squared bishop and preparing to castle kingside.

<|user|>: 9.0-0-0

<|bot|>: With 9.0-0-0, you are now castling kingside and putting your king in a safer position. This move also has the idea of bringing the rook into the game more quickly. As black, I will castle kingside as well with 9...0-0-0.

<|user|>: 10.Kb1

<|bot|>: I'm sorry, but that is an illegal move. In chess, the king can only move one square at a time and cannot "jump" over other pieces. The correct move would be 10.Be2, developing your bishop and preparing to castle kingside.

<|endoftext|>"
}

## 中文用户请看这里
为推进中文人工智能产业发展，促进AI技术的公开化与国际化，我们发起了ShareGPT-90k项目，旨在凝聚各界力量开展数据清洗与对齐工作。

与多数人的认知不同，GPT模型的性能主要依托预训练语料库实现，语料质量对模型最终性能起着决定性作用。然而当前百度知道、CSDN、知乎等平台低质营销软文泛滥，小木虫等高质量平台的语料资源相对匮乏，个人博客内容质量更是良莠不齐。

OpenAI在数据集收集过程中耗费了巨额成本，甚至需要向微软募资。鉴于我们无力承担如此高昂的开销，因此恳请有志于构建开放获取语料库且具备一定外语能力的网友贡献力量。

若您有意参与本项目，我们将不胜感激。
项目地址：https://paratranz.cn/projects/6725

原始信息汇总

Multilingual Share GPT 数据集概述

数据集名称

• 名称: Multilingual Share GPT
• 描述: 一个免费的多语言语料库，用于LLM训练。所有文本已转换为Markdown格式，并按语言分类。

数据集内容

• 语言: 包含英语（en）和中文（zh）。
• 示例: json { "id": "ImiMfCY", "lang": "en", "text": " <|user|>: Lets play chess

<|bot|>: Sure, Id love to play chess with you! Do you want to play a virtual game or just ask chess-related questions?

<|user|>: Virtual game

<|bot|>: Im sorry, Im an AI language model and I dont have the ability to display a chess board or move pieces on it. However, Id be happy to help you with any chess-related questions or problems you might have. Just let me know what you need help with!

<|user|>: Lets play

<|bot|>: I apologize for any confusion, but as an AI language model I do not have the ability to play chess as it requires a graphical interface. However, I can help you learn and improve your chess skills by answering questions and providing explanations and analysis of specific positions or games.

<|user|>: 1.e4

<|bot|>: The move 1.e4 is the most common first move in the game of chess, and is known as the "Kings Pawn Opening". This move aims to control the center of the board and create space for the other pieces to develop. Black has a variety of responses to 1.e4, but some of the most common are 1...e5 (the "Kings Gambit"), 1...c5 (the "Sicilian Defense"), and 1...e6 (the "French Defense").

<|user|>: I play 1.e4

<|bot|>: Very good! You have started the game with the Kings Pawn Opening. As black, I will respond with 1...e5, the Kings Gambit.

<|user|>: I play 2.Nf3

<|bot|>: Thats a good move. With 2.Nf3, you are developing your knight and attacking the black pawn on e5. As black, Ill play 2...Nc6, which develops the knight to a strong central square and prepares to castle kingside.

<|user|>: I play 3.d4

<|bot|>: With the move 3.d4, you are now offering a pawn to black with the Kings Gambit. This gambit can lead to a sharp and tactical game, but it also weakens the white pawn structure and can lead to a disadvantage if not handled properly. As black, I will accept the gambit with 3...exd4.

<|user|>: I play 4.Nxd4

<|bot|>: With 4.Nxd4, you are now attacking the black pawn on d4 and forcing it to capture. As black, Ill capture the pawn with 4...Nxd4. This has now opened up the d-file for my rook and given me a lead in development.

<|user|>: I play 5.Qxd4

<|bot|>: With the move 5.Qxd4, you have taken control of the d-file and are attacking the black knight on d4. As black, I will respond with 5...Nc6, developing my knight to a safer square and putting pressure on the white queen.

<|user|>: I play 6.Qd3

<|bot|>: With 6.Qd3, you are attacking the black knight again and preparing to castle kingside. This move also has the idea of putting pressure on the d7-pawn, as well as the d8-h4 diagonal. As black, I will respond with 6...Nf6, developing my knight to a strong central square and preparing to castle kingside.

<|user|>: I play 7.Nc3

<|bot|>: With 7.Nc3, you are developing your knight and preparing to castle kingside. This move also has the idea of supporting the d4-queen and putting pressure on the e5-pawn. As black, I will play 7...d6, solidifying my pawn structure and preparing to develop my dark-squared bishop.

<|user|>: I play 8.Bg5

<|bot|>: With 8.Bg5, you are attacking the black knight on f6 and preparing to castle kingside. This move also has the idea of putting pressure on the h7-pawn and controlling the f4-square. As black, I will respond with 8...Be7, developing my dark-squared bishop and preparing to castle kingside.

<|user|>: I play 9.0-0-0

<|bot|>: With 9.0-0-0, you are now castling kingside and putting your king in a safer position. This move also has the idea of bringing the rook into the game more quickly. As black, I will castle kingside as well with 9...0-0-0.

<|user|>: I play 10.Kb1

<|bot|>: Im sorry, but that is an illegal move. In chess, the king can only move one square at a time and cannot "jump" over other pieces. The correct move would be 10.Be2, developing your bishop and preparing to castle kingside.

<|endoftext|>" }

许可证

• 许可证: CC0-1.0

搜集汇总

数据集介绍

构建方式

该数据集名为 Multilingual Share GPT，旨在为大型语言模型训练提供免费的多语言语料库。其构建方式基于社区贡献与自动化处理相结合：所有文本被统一转换为 Markdown 格式，并按语言进行分类，确保数据结构的标准化与可扩展性。数据样例呈现为 JSON 格式，包含唯一标识符、语言标签及对话文本，其中对话采用用户与助手的交互模式，覆盖多样化的场景，如国际象棋对弈等，以模拟真实的多轮对话。这种构建策略降低了数据收集的门槛，通过开放平台（如 GitHub 和 Paratranz）汇聚志愿者力量，实现了去中心化的语料积累与清洗。

使用方法

使用该数据集时，用户可直接从 HuggingFace 页面下载 JSON 格式的文件，并按照语言标签（如 en、zh）筛选所需语料。数据可直接用于监督微调或预训练阶段，通过解析对话结构中的角色标识，构建输入-输出对以训练模型的对话生成能力。推荐结合标准 NLP 工具进行进一步清洗，例如去除冗余标记或平衡语言分布。开发者也可参照 GitHub 仓库中的示例，将数据适配至自定义训练管线，或通过 Paratranz 平台参与后续标注与扩充，以增强数据集的领域覆盖度。

背景与挑战

背景概述

在大型语言模型（LLM）的研发浪潮中，高质量多语种语料库的匮乏始终是制约模型泛化能力与跨语言理解性能的关键瓶颈。cryscan/multilingual-share数据集由开源社区于2023年发起，核心研究机构与个人贡献者依托RWKV-Wiki项目，旨在构建一个免费、开放的多语言语料集，专门用于LLM的训练与对齐。该数据集涵盖了英语与中文等多种语言，所有文本均统一转换为Markdown格式，并按照语言进行分类，显著降低了多语言数据处理的复杂性。其核心研究问题在于如何通过社区协作的方式，高效收集并清洗分散于互联网中的优质语料，以弥补现有数据集（如百度知道、知乎等平台）中软文泛滥、高质量内容稀缺的缺陷。该数据集对相关领域的影响力体现在其推动了AI技术的公开化与国际化，为全球研究者提供了可复用的基础资源，尤其对中文LLM的发展具有里程碑式的意义。

当前挑战

当前，cryscan/multilingual-share数据集面临多重挑战。首先，在领域问题层面，它致力于解决多语言LLM训练中语料质量参差不齐的顽疾。现有平台如百度知道、CSDN充斥着大量软文，而小木虫等高质量平台语料稀缺，个人博客内容良莠不齐，导致模型在跨语言任务中容易受到噪声干扰，难以习得精准的语义表征。其次，在构建过程中，数据集遭遇了资源与协作上的严峻考验。OpenAI曾为语料收集耗费巨额成本，而该开源项目缺乏类似财力支持，需依赖社区志愿者的力量进行数据清洗与对齐，但外语基础扎实、有意愿参与的高质量贡献者数量有限，使得语料扩展与质量控制进度缓慢。此外，不同语言间文化背景与表达习惯的差异，进一步增加了数据标准化与标注一致性的难度。

常用场景

经典使用场景

Multilingual Share GPT 数据集专为大规模语言模型（LLM）的多语种预训练与指令微调而设计，其核心场景在于构建双语或多语对话系统。该数据集以中英文对话为核心，采用统一的 Markdown 格式，并标注语言类别，使得研究者能够轻松提取高质量的多轮交互语料，用于训练模型在多语言环境下理解上下文、生成自然回复，尤其适用于跨语言问答、翻译辅助对话及文化适应型聊天机器人的开发。

解决学术问题

该数据集直面当前学术界在开源多语种语料库领域的稀缺性难题，尤其是中文高质量对话数据的不足。传统语料如百度知道、CSDN 等平台充斥软文，而小木虫等专业论坛数据量有限，导致模型在中文场景下表现欠佳。Multilingual Share GPT 通过社区协作清洗与对齐，提供了免费、开放的多语种对话数据，解决了预训练阶段语料质量参差不齐的问题，为研究多语言模型的对齐性、泛化能力及文化偏见消解提供了坚实的数据基础。

实际应用

在实际应用中，该数据集可赋能企业级多语种客服系统、教育领域的语言学习辅助工具，以及面向全球用户的智能助手。例如，基于该数据训练的模型能流畅处理中英文混合输入，在电商客服中实现跨语言订单查询，或在语言学习平台中提供实时对话练习。此外，其开放许可（CC0-1.0）降低了商业部署的法律风险，加速了从研究到产品的转化进程。

数据集最近研究

最新研究方向

当前多语言语料库的研究前沿聚焦于构建高质量、开放获取的平行语料，以支撑大规模语言模型（LLM）的预训练与对齐优化。cryscan/multilingual-share数据集作为Multilingual Share GPT项目的核心资源，采用CC0-1.0许可协议，收录中英双语且经Markdown格式规范化的对话文本，其设计理念契合了开源社区对透明化、低成本数据构建的迫切需求。该数据集通过社区协作与人工清洗机制，规避了传统语料中软文泛滥、质量参差的痛点，尤其在与OpenAI等机构高昂数据采集成本的对比下，凸显了众包模式在推动AI技术民主化进程中的关键作用。这一研究方向不仅响应了国际间对多语言模型公平性的呼吁，也为中文AI的全球化发展提供了可复用的基础设施，其影响深远，标志着高质量语料库从封闭垄断向开放共享的范式转变。