ewof/hf-instruction-unfiltered

This dataset is HuggingFaceH4/instruction-dataset unfiltered, removing 10 instances of blatant alignment.
317 instructions remain.

clean.py was first ran on https://huggingface.co/datasets/HuggingFaceH4/instruction-dataset/blob/fc50699d275ed86a072f1166f7390473176719fd/step3-eval.jsonl then renamed to .json not .jsonl

normal dedupe.py script didn't find any dupes here.

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py

原始信息汇总

数据集概述

数据集名称

HuggingFaceH4/instruction-dataset

数据集大小

• 原始数据集：包含317条指令。
• 处理后数据集：移除了10条明显不符合要求的指令。

数据集处理

• 使用clean.py脚本对数据集进行了清洗。
• 数据集文件从.jsonl格式转换为.json格式。

数据集去重

• 通过dedupe.py脚本进行了去重处理，未发现重复数据。

数据集灵感来源

• 灵感来源于ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered数据集。

数据集清理脚本

• 清理脚本wizardlm_clean.py和clean.py均由anon8231489123提供的脚本改编而来。