Chain-of-Thought collection

Dataset accompanying the paper "The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuning", including 1.88M CoT rationales extracted across 1,060 tasks" - https://arxiv.org/abs/2305.14045

**From the release repo** https://github.com/kaistAI/CoT-Collection: Large Language Models (LLMs) have shown enhanced capabilities of solving novel tasks by reasoning step-by-step known as Chain-of-Thought (CoT) reasoning; how can we instill the same capability of reasoning step-by-step on unseen tasks into LMs that possess less than <100B parameters? To address this question, we first introduce the CoT Collection, a new instruction-tuning dataset that augments 1.88 million CoT rationales across 1,060 tasks. We show that continually fine-tuning Flan-T5 (3B & 11B) with the CoT Collection enables the 3B & 11B LMs to perform CoT better on unseen tasks, leading to an improvement in the average zero-shot accuracy on 27 datasets of the BIG-Bench-Hard benchmark by +4.34% and +2.44%, respectively. Furthermore, we show that instruction tuning with CoT allows LMs to possess stronger few-shot learning capabilities, resulting in an improvement of +2.97% and +2.37% on 4 domain-specific tasks over Flan-T5 (3B & 11B), respectively.

随论文《The CoT Collection: 通过思维链微调提升大语言模型的零样本和少样本学习》附带的语料库，包含跨越1,060项任务的188万条思维链（CoT）解释。该语料库旨在通过增强Flan-T5（3B & 11B参数规模）模型，赋予其逐步推理的能力，以解决参数规模低于100亿的未见过的新任务。为此，我们首先提出了CoT Collection，这是一个新的指令微调数据集，它丰富了1.88百万条思维链解释。研究表明，通过持续微调Flan-T5模型，能够显著提升其在未见任务上的思维链推理能力，进而使得3B和11B参数规模的模型在BIG-Bench-Hard基准测试的27个数据集上的零样本平均准确率分别提升了4.34%和2.44%。此外，我们还发现，结合思维链的指令微调能够增强大语言模型的少样本学习能力，相对于Flan-T5（3B & 11B），在4个特定领域的任务上分别提升了2.97%和2.37%。