cfahlgren1/Capybara-Converted

---
license: apache-2.0
task_categories:
- conversational
- question-answering
- text-generation
language:
- en
tags:
- Physics
- Biology
- Math
- Chemistry
- Culture
- Logic
- Roleplay
pretty_name: LessWrong-Amplify-Instruct
size_categories:
- 10K<n<100K
---
## This is the Official Capybara dataset. Over 10,000 multi-turn examples.

Capybara is the culmination of insights derived from synthesis techniques like Evol-instruct (used for WizardLM), Alpaca, Orca, Vicuna, Lamini, FLASK and others.
The single-turn seeds used to intiate the Amplify-Instruct synthesis of conversations are mostly based on datasets that i've personally vetted extensively, and are often highly regarded for their diversity and demonstration of logical robustness and prose, such as Airoboros, Know logic, EverythingLM, GPTeacher and even entirely new seed instructions derived from different sources, including certain in-house multi-turn datasets like Dove and Verified-Camel(A successor to Puffin).

The multi-turn synthetic conversation generation method is what i'm calling Amplify-Instruct, and the first resulting dataset using this method is called Capybara.
This dataset has a strong focus on information diversity across a wide range of domains, and multi-turn conversations that strongly emphasize reasoning, logic and extrapolation about a wide range of subjects, also many great examples of conversations delving into obscure sub-topics and rabbit holes across pop-culture and STEM, while also maintaining natural prose.
While performing great in it's current state, the current dataset used for fine-tuning is entirely contained within 20K training examples, this is 10 times smaller than many similar performing datasets, this is signficant when it comes to scaling implications once I decide to scale the use of Amplify-Instruct to significantly more examples.

- Most tokens contained in this dataset are newly synthesized and did not exist prior online.

- This leverages the Amplify-Instruct method(paper coming soon) to grow thousands of high-quality single-turn seeds into advanced and in-depth multi-turn conversations.

- Average context length per conversation is over 1,000 tokens and 3 turns or more per example (most instruction/chat datasets on HF for fine-tuning are only 1 turn)

- Each conversation is optimized to amplify the natural raw knowledge capabilities of the model, as well as delving deep into obscure and advanced topics.

- Aggresively filtered to remove any and all possible examples of overt moralizing/alignment, and common undesirable behaviours such as "as an AI language model" and "September 2021" and "I don't have personal beliefs"

## Benchmarks.

- Resulting benchmarks are available on HF Leaderboard, and other benchmarks done as well such as AGIEval, Bigbench and GPT4All.
- (The only Capybara model available on all of these benchmarks including HF leaderboard is Capybara V1, trained on Llama-2)
- The below benchmarks are compared against fine-tunes also done on Llama-2.



## Quality filtering and cleaning.

- Extensive measures were done to filter out any conversations that contained even a single instance of overt AI moralizing/alignment, such as "As an AI language model" and common undesirable behaviours such as conversations that include "September 2021" and "I don't have personal beliefs" and other phrases I've found to be highly correlated with undesirable responses and conversation paths.

## Thank you to those of you that have indirectly contributed!

While most of the tokens within Capybara are newly synthsized and part of datasets like Puffin/Dove, we would like to credit the single-turn datasets we leveraged as seeds, which were used to generate the multi-turn data.

The datasets shown in green below are datasets that we sampled from to curate seeds that are used during Amplify-Instruct synthesis for this project, however, most of the tokens in capybara within those given sections are novel tokens not present in any of the seed datasets.

Datasets in Blue are in-house curations that previously existed prior to Capybara, and were now used as seeds for Capybara.


## Dataset contamination.

We have checked the capybara dataset for contamination for several of the most popular benchmarks and can confirm that there is no contaminaton found besides MT-bench which is now cleaned out.

We leveraged minhash to check for 100%, 99%, 98% and 97% similarity matches between our data and the questions and answers in benchmarks, we found no exact matches, nor did we find any matches down to the 97% similarity level.

The following are benchmarks we checked for contamination against our dataset:

- HumanEval

- AGIEval

- TruthfulQA

- MMLU

- GPT4All

*Newly cleaned out as of 12/15/2023 - MT-bench
## Credits
During the curation process, there can be some relatively arduos steps when it comes to actually executing on the best experimentation or concepts for how to filter examples out.
Luckily there is folks over at Nous Research that helped with expediting these processes, big thank you to J-Supha specifically for making these types of significant contributions.
## Example Outputs from the Llama-2 7B model trained on this dataset:
## Future Plans & How you can help!
This is a relatively early build amongst the grand plans for the future of what I plan to work on!
In the near future we plan on leveraging the help of domain specific expert volunteers to eliminate any mathematically/verifiably incorrect answers from training curations of different types of datasets.
If you have at-least a bachelors in mathematics, physics, biology or chemistry and would like to volunteer even just 30 minutes of your expertise time, please contact LDJ on discord!
Citation:
```
@article{daniele2023amplify-instruct,
title={Amplify-Instruct: Synthetically Generated Diverse Multi-turn Conversations for Effecient LLM Training.},
author={Daniele, Luigi and Suphavadeeprasit},
journal={arXiv preprint arXiv:(coming soon)},
url={https://huggingface.co/datasets/LDJnr/Capybara},
year={2023}
}
```

原始信息汇总

数据集概述

基本信息

• 许可证: Apache-2.0
• 任务类别:
  • 对话
  • 问答
  • 文本生成
• 语言: 英语
• 标签:
  • 物理
  • 生物
  • 数学
  • 化学
  • 文化
  • 逻辑
  • 角色扮演
• 数据集名称: LessWrong-Amplify-Instruct
• 数据集大小: 10K<n<100K

数据集描述

• 数据集来源: 基于多种合成技术（如Evol-instruct、Alpaca、Orca等）的综合洞察。
• 种子数据: 主要基于经过严格审查的数据集，如Airoboros、Know logic、EverythingLM等。
• 合成方法: 采用名为Amplify-Instruct的方法，将高质量的单轮种子扩展为深入的多轮对话。
• 数据特点:
  • 强调推理、逻辑和跨学科主题的扩展。
  • 包含大量涉及流行文化和STEM领域的深奥子话题的对话。
  • 平均每个对话超过1,000个令牌和3轮以上。
• 质量控制:
  • 严格过滤，移除任何可能的道德化/对齐示例和常见的不良行为。
  • 使用minhash检查与多个基准的相似性，确保无污染。

未来计划

• 计划利用领域专家志愿者来消除训练数据中的数学/可验证错误答案。

引用

@article{daniele2023amplify-instruct, title={Amplify-Instruct: Synthetically Generated Diverse Multi-turn Conversations for Effecient LLM Training.}, author={Daniele, Luigi and Suphavadeeprasit}, journal={arXiv preprint arXiv:(coming soon)}, url={https://huggingface.co/datasets/LDJnr/Capybara}, year={2023} }

搜集汇总

数据集介绍

构建方式

Capybara数据集的构建根植于对多种先进合成技术的深刻洞察，如Evol-instruct、Alpaca、Orca等。其核心方法名为Amplify-Instruct，通过将经过严格筛选的单轮种子对话（源自Airoboros、Know logic等高质量数据集）扩展为深入的多轮交互。每一轮对话均经过优化，以放大模型对自然知识的原始理解能力，并深入探索晦涩与前沿主题。该过程生成了超过10,000个多轮样本，其中大部分token为全新合成，不存在于现有网络中。

特点

该数据集以信息多样性为核心，横跨物理、生物、数学、化学、文化、逻辑及角色扮演等多个领域。其显著特点在于平均每段对话超过1,000个token，且包含至少三轮交互，远超常见的单轮指令数据集。通过激进的质量过滤，数据集剔除了任何显性道德化或对齐性表述（如“作为一个AI语言模型”），以及“2021年9月”等不良行为模式。此外，其训练集仅含20,000个样本，却展现出与十倍规模数据集相媲美的性能，凸显了高效扩展的潜力。

使用方法

Capybara数据集适用于对话系统、问答及文本生成任务。用户可通过HuggingFace平台直接加载，例如使用`datasets.load_dataset('cfahlgren1/Capybara-Converted')`。该数据集已针对主流基准（如HumanEval、MMLU）进行污染检测，确认无显著重叠。推荐基于Llama-2等模型进行微调，以充分利用其多轮推理与逻辑延展能力。开发者可参考随附的Amplify-Instruct论文（即将发布）以复现或改进合成流程。

背景与挑战

背景概述

Capybara数据集由Luigi Daniele与Suphavadeeprasit于2023年创建，依托Nous Research等机构的协作，旨在解决大语言模型在多轮对话中推理深度与领域多样性的不足。该数据集融合了Evol-instruct、Alpaca、Orca等合成技术的精华，通过创新的Amplify-Instruct方法，将数千条高质量单轮种子指令扩展为超过一万条多轮对话样本，平均上下文长度逾千词元，覆盖物理、生物、数学、化学、文化、逻辑及角色扮演等广泛领域。其核心研究问题在于探索如何在有限训练样本（仅2万条）下，通过数据合成提升模型的逻辑推理与知识泛化能力。Capybara在HuggingFace排行榜及AGIEval、Bigbench等基准测试中表现优异，且经检测与HumanEval、MMLU等主流基准无污染，凸显了其在多轮对话数据构建领域的创新性与影响力。

当前挑战

Capybara数据集面临的主要挑战包括：首先，在领域问题层面，现有大语言模型在多轮对话中常陷入浅层回应或知识窄化，而Capybara需解决如何通过合成数据增强模型对晦涩主题的深入推理与跨领域衔接能力，同时避免过度对齐（如“作为AI语言模型”等表述）导致的自然性损失。其次，在构建过程中，数据合成面临质量控制的严峻考验——需从Airoboros、Know Logic等种子集中筛选高多样性指令，并通过Amplify-Instruct方法生成多轮对话，但需剔除数学或逻辑错误，且当前仅2万条示例的规模远小于同类数据集（如WizardLM的数十万条），未来扩展至更大规模时需解决合成效率与一致性难题。此外，数据集还面临基准污染检测的复杂性，尽管已清理MT-bench，但持续维护无污染状态仍是长期挑战。

常用场景

经典使用场景

Capybara数据集专为多轮对话生成与指令微调而设计，广泛应用于提升大语言模型在复杂推理、逻辑推演及跨领域知识整合方面的能力。其经典使用场景涵盖从物理、数学到文化、角色扮演等多元主题的深度对话生成，尤其强调模型在长上下文（平均超1000个token）下保持自然语流与信息连贯性。通过Amplify-Instruct方法，该数据集将单轮高质量种子指令扩展为多轮交互，成为训练如Llama-2等模型在逻辑鲁棒性与知识深度上突破的基石。

实际应用

在实际应用中，Capybara赋能大模型在智适应教育、科学问答助手及创意写作等场景中展现卓越表现。例如，基于该数据集微调的模型能够深入探讨物理悖论、化学机理等专业主题，并生成逻辑自洽的角色扮演对话。其过滤策略也使得模型在避免过度对齐的同时保持安全边界，适用于需要开放性与专业性的商业客服、虚拟导师及内容创作工具。

衍生相关工作

Capybara衍生了一系列影响深远的工作，包括其前身Puffin与Dove数据集，以及基于Amplify-Instruct方法的新一代多轮对话库。该数据集催生了针对小样本高效微调的研究，并启发了如Verified-Camel等后续项目，进一步验证了合成数据在提升模型推理能力上的潜力。相关基准评测（如AGIEval、BigBench）的优异表现，也推动了社区对数据质量与多样性优先于规模的共识。