PhilipMay/UltraChat-200k-ShareGPT-clean

---
license: mit
language:
- en
size_categories:
- 100K<n<1M
---

# UltraChat-200k ShareGPT Clean

This dataset is cleaned and created with [`01_convert_ultrachat_200k_train_sft.ipynb`](https://github.com/PhilipMay/llm-data/blob/main/01_convert_ultrachat_200k_train_sft.ipynb) and
[`02_convert_ultrachat_200k_test_sft.ipynb`](https://github.com/PhilipMay/llm-data/blob/main/02_convert_ultrachat_200k_test_sft.ipynb)
based on [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k)
(train_sft and test_sft).

Main changes:

- convert to conversations format which is supported by [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) - see [ShareGPT](https://github.com/OpenAccess-AI-Collective/axolotl?tab=readme-ov-file#conversation)
- clean invisible characters and strip - see
[`mltb2.text.clean_all_invisible_chars_and_strip()`](https://telekom.github.io/mltb2/api-reference/text.html#mltb2.text.clean_all_invisible_chars_and_strip)
- remove rows with empty text

## Licensing

Copyright (c) 2024 [Philip May](https://philipmay.org/)\
Copyright (c) The Hugging Face team\
Copyright (c) THUNLP

Licensed under the **MIT License** (the "License"); you may not use this file except in compliance with the License.
You may obtain a copy of the License by reviewing the file
[LICENSE](https://huggingface.co/datasets/PhilipMay/UltraChat-200k-ShareGPT/blob/main/LICENSE) in the repository.

原始信息汇总

UltraChat-200k ShareGPT Clean 数据集概述

数据集基本信息

• 名称: UltraChat-200k ShareGPT Clean
• 语言: 英语（en）
• 大小: 10万至100万条记录
• 许可证: MIT License

数据集创建与处理

• 创建工具: 使用 01_convert_ultrachat_200k_train_sft.ipynb 和 02_convert_ultrachat_200k_test_sft.ipynb 基于 HuggingFaceH4/ultrachat_200k 数据集进行处理。
• 主要变更:
  • 转换为 Axolotl 支持的对话格式。
  • 清理不可见字符并去除多余空格，使用 mltb2.text.clean_all_invisible_chars_and_strip() 方法。
  • 移除文本为空的行。

版权信息

• 版权所有者: Philip May, The Hugging Face team, THUNLP
• 许可证: MIT License，详细内容可在 LICENSE 文件中查看。