ewof/dolly-instruct-unfiltered-deduped

This dataset is databricks/databricks-dolly-15k unfiltered and deduped, removing 640 instances of blatant alignment and 14 duplicates.
14357 instructions remain.

clean.py was first ran on https://huggingface.co/datasets/databricks/databricks-dolly-15k/blob/d72c16e4644a463b9c678c71d9440befd4594556/databricks-dolly-15k.jsonl and then dedupe.py was ran on it. renamed to .json not .jsonl

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py

原始信息汇总

数据集概述

数据集名称

• 名称: databricks/databricks-dolly-15k

数据集处理

• 原始数据: 包含15000条记录
• 处理步骤:
  • 使用clean.py脚本进行初步清理
  • 使用dedupe.py脚本进行去重处理
• 处理结果:
  • 移除了640条明显的对齐数据和14条重复记录
  • 剩余数据量: 14357条指令

数据格式

• 原始格式: .jsonl
• 最终格式: .json

数据集灵感来源

• 灵感来源于: https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

清理脚本来源

• 清理脚本clean.py改编自anon8231489123提供的脚本，并进行了适应性修改。