HieraDate

已经提出了几种多跳阅读理解数据集来解决推理快捷方式的问题，通过这些快捷方式可以在不执行多跳推理的情况下回答问题。但是，在找到比较问题的答案时，多跳模型执行逐步推理的能力尚不清楚。还不清楚有关内部推理过程的问题如何用于培训和评估问题回答 (QA) 系统。为了以分层的方式精确评估模型，我们首先提出了一个数据集，HieraDate，除了主要问题之外，还有三个探测任务: 提取，推理和鲁棒性。我们的数据集是通过增强之前的两个多跳数据集HotpotQA和2WikiMultiHopQA创建的，重点是关于日期信息的多跳问题，涉及比较和数字推理。然后，我们评估现有模型理解日期信息的能力。我们的实验结果表明，即使多跳模型在日期比较和数字减法任务中表现良好，它们也无法减去两个日期。其他结果表明，我们的探测问题可以帮助提高主要QA任务上模型的性能 (例如，通过10.3 F1)，我们的数据集可以用于数据增强，以提高模型的鲁棒性。Several multi-hop reading comprehension datasets have been proposed to address the issue of reasoning shortcuts, through which questions can be answered without performing multi-hop reasoning. However, the ability of multi-hop models to execute step-by-step reasoning when seeking answers to comparative questions remains unclear. It is also unclear how questions targeting the internal reasoning process can be utilized to train and evaluate question answering (QA) systems. To accurately evaluate models in a hierarchical manner, we first propose a dataset named HieraDate, which includes three probe tasks alongside the main question: extraction, reasoning, and robustness. Our dataset is constructed by augmenting two existing multi-hop datasets, HotpotQA and 2WikiMultiHopQA, with a focus on multi-hop questions related to date information, involving comparison and numerical reasoning. We then evaluate the capability of existing models to understand date information. Our experimental results indicate that even though multi-hop models achieve strong performance on date comparison and numerical subtraction tasks, they fail to correctly subtract two dates. Additional findings show that our probe questions can help boost model performance on the main QA task (e.g., improving F1 score by 10.3), and our dataset can be employed for data augmentation to enhance model robustness.

搜集汇总

背景与挑战

背景概述

HieraDate是一个基于HotpotQA和2WikiMultiHopQA构建的多跳阅读理解数据集，专注于日期比较和数字推理问题，包含提取、推理和鲁棒性三个探测任务。该数据集旨在评估模型的分层推理能力，并可通过数据增强提升模型在主要QA任务上的性能。