TMLR-Group-HF/NoRa

NoRa（带噪声推理）数据集是一种专门设计的数据集，用于评估大型语言模型在面对带噪声推理过程时的推理能力。该数据集包含带有干净推理样本和不同类型及难度噪声的推理任务。数据集根据三个主要属性组织：任务类型（包括数学运算、符号操作和常识推理等）、噪声类型（包括干净、不相关和不准确）、难度等级（包括简单、中等、困难和无噪声）。每个样本包含问题文本、正确答案、任务类型、噪声类型、难度等级、示例问题和答案的链式思维演示、每个演示中的平均思维步骤数、每个演示中的平均噪声思维步骤数以及噪声分布类型。

NoRa (Noisy Rationales) is a dataset specifically designed to evaluate the reasoning capabilities of Large Language Models (LLMs) when faced with noisy reasoning processes. The dataset contains reasoning tasks with both clean reasoning samples and samples with different types and difficulties of noise. The dataset structure is based on three main attributes: task types (including mathematical operations, symbolic manipulation, and commonsense reasoning), noise types (including clean, irrelevant, and inaccurate), and difficulty levels (including easy, medium, hard, and none for clean samples). Each sample includes the question text, the correct answer, the task type, the noise type, the difficulty level, chain-of-thought demonstrations with example questions and answers, the average number of thinking steps in each demonstration, the average number of noisy thinking steps in each demonstration, and the type of noise distribution.