ewof/self-instruct-unfiltered-deduped
收藏Hugging Face2023-10-01 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/ewof/self-instruct-unfiltered-deduped
下载链接
链接失效反馈官方服务：
资源简介：
This dataset is https://github.com/yizhongw/self-instruct unfiltered and deduped, removing 1600 instances of blatant alignment and 26 duplicates.
80813 instructions remain.
clean.py was first ran on https://github.com/yizhongw/self-instruct/blob/0b26ccaa415992100fa32df62d41b994cf928e23/data/gpt3_generations/batch_221203/all_instances_82K.jsonl and then dedupe.py was ran on it, renamed to .json not .jsonl
inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered
All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean.py
提供机构：
ewof
原始信息汇总
数据集概述
数据集来源与处理
- 数据集源自
https://github.com/yizhongw/self-instruct，经过去重和过滤处理。 - 移除了1600个明显的对齐实例和26个重复项。
- 最终保留了80813条指令。
数据处理脚本
clean.py脚本首先在https://github.com/yizhongw/self-instruct/blob/0b26ccaa415992100fa32df62d41b994cf928e23/data/gpt3_generations/batch_221203/all_instances_82K.jsonl上运行。- 随后使用
dedupe.py脚本进行去重处理，并将文件格式从.jsonl更改为.json。
灵感来源
- 数据集的灵感来源于
https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered。
脚本贡献者
- 清理脚本由 anon8231489123 提供，原作者将其改编为
wizardlm_clean.py，后进一步改编为clean.py。
