ewof/oasst-convo-unfiltered-deduped

This dataset is the Open Assistant dataset https://huggingface.co/datasets/OpenAssistant/oasst1, and formatted to be easier to convert to whatever finetune data format you want, removes 75 instances of alignment and 81 dupes. oasst_clean_format_dedupe.py was first ran on 2023-04-12_oasst_all.trees.jsonl from OpenAssistant/oasst1 Inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered Credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to oasst_clean_data_format.py I converted to trees so each object in the output has a messages list, each message has a role and content. Each starting prompt in the oasst dataset will appear in the output n times where n is the number of replies the prompt had (unless it had a duplicate). so if the oasst tree looked like ``` user: aaa assistant: bbb user: ccc assistant: ddd assistant: eee user: fff assistant: ggg user: hhh assistant: iii ``` then the output would look like ``` { messages: [ { "role": "user", "content": "aaa" }, { "role": "model", "content": "bbb" }, { "role": "user", "content": "ccc" }, { "role": "model", "content": "ddd" } ] } { messages: [ { "role": "user", "content": "aaa" }, { "role": "model", "content": "eee" } // if tree ends with a user msg then it is not included in the output, this can be changed by passing --save-user-ends to the script ] } { messages: [ { "role": "user", "content": "aaa" }, { "role": "model", "content": "ggg" }, { "role": "user", "content": "hhh" }, { "role": "model", "content": "iii" } ] } ```

原始信息汇总

数据集概述

数据集名称

• OpenAssistant/oasst1

数据集来源

• 原始数据集链接：OpenAssistant/oasst1

数据处理

• 数据集经过oasst_clean_format_dedupe.py脚本处理，移除了75个对齐实例和81个重复项。
• 处理日期：2023-04-12

数据格式

• 数据转换为树状结构，每个输出对象包含一个messages列表。
• 每个消息包含role和content两个属性。
• 原始数据集中的每个起始提示将根据其收到的回复次数（排除重复）在输出中出现相应次数。

示例输出

json { "messages": [ { "role": "user", "content": "aaa" }, { "role": "model", "content": "bbb" }, { "role": "user", "content": "ccc" }, { "role": "model", "content": "ddd" } ] } { "messages": [ { "role": "user", "content": "aaa" }, { "role": "model", "content": "eee" } ] } { "messages": [ { "role": "user", "content": "aaa" }, { "role": "model", "content": "ggg" }, { "role": "user", "content": "hhh" }, { "role": "model", "content": "iii" } ] }