ewof/flan_unfiltered

tsvs are from here https://github.com/google-research/FLAN/tree/4c5f0ba64239f7ff98fe90a5426fa175f4d1713c/flan/v2/cot_data
1 dupe removed, 31356 instances of alignment removed

dataset preview is fucked, it looks like this
```
{"question": "What does the government have control over?\\nOptions:\\n- trouble\\n- country\\n- army\\n- city\\n- control", "answer": "city", "chain_of_thought": "A city is a large town. A government controls large towns.\n"}
```

inspired by https://huggingface.co/datasets/ehartford/WizardLM_alpaca_evol_instruct_70k_unfiltered

All credit to anon8231489123 for the cleanup script that I adapted to wizardlm_clean.py, I then took this script and adapted it to clean_format_dedupe.py

原始信息汇总

数据集概述

数据来源

• 数据集的tsv文件来源于GitHub仓库：https://github.com/google-research/FLAN/tree/4c5f0ba64239f7ff98fe90a5426fa175f4d1713c/flan/v2/cot_data

数据处理

• 已移除1个重复实例
• 移除了31356个对齐实例

数据示例

• 数据集预览示例： json { "question": "What does the government have control over?\nOptions:\n- trouble\n- country\n- army\n- city\n- control", "answer": "city", "chain_of_thought": "A city is a large town. A government controls large towns. " }