ewof/gpteacher-unfiltered

This dataset is https://github.com/teknium1/GPTeacher unfiltered, removing 1489 instances of blatant alignment.
23073 instructions remain.

https://github.com/teknium1/GPTeacher/blob/8afcaaa7a11dd980162d861bd6be970f95eb7174/Codegen/codegen-instruct.json

https://github.com/teknium1/GPTeacher/blob/e3b7aba886c6c0c8ad30a650edfa7a3093fbf57c/Instruct/gpt4-instruct-dedupe-only-dataset.json

https://github.com/teknium1/GPTeacher/blob/5b040645528a38bfa81a258e7646f8c92ad7d0dd/Roleplay/roleplay-simple-deduped-roleplay-instruct.json

i combined all of these files above into gpteacher.json and ran clean.py

normal dedupe.py script didn't find any dupes here.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py

原始信息汇总

数据集概述

数据集名称

• GPTeacher

数据集描述

• 该数据集经过处理，移除了1489个明显不相关的实例，剩余23073条指令。

数据集文件

• codegen-instruct.json
  • 存储位置：GitHub链接
• gpt4-instruct-dedupe-only-dataset.json
  • 存储位置：GitHub链接
• roleplay-simple-deduped-roleplay-instruct.json
  • 存储位置：GitHub链接

数据集处理

• 所有上述文件已合并为gpteacher.json，并执行了clean.py脚本进行清理。
• dedupe.py脚本未在此数据集中发现重复内容。

数据集来源与灵感

• 灵感来源于Hugging Face数据集。
• 清理脚本由anon8231489123提供，原作者将其适配为wizardlm_clean.py，后被适配为clean.py。