hfl/expmrc
收藏资源简介：
--- license: cc-by-sa-4.0 task_categories: - question-answering language: - zh - en --- ## GitHub repository: https://github.com/ymcui/expmrc With the development of the pre-trained language models (PLMs), achieving human-level performance on several machine reading comprehension (MRC) dataset is not as hard as it used to be. However, the explainability behind these artifacts still remains unclear, raising concerns on utilizing these models in real-life applications. To improve the explainability of MRC tasks, we propose ExpMRC benchmark. **ExpMRC** is a benchmark for **Exp**lainability Evaluation of **M**achine **R**eading **C**omprehension. ExpMRC contains four subsets of popular MRC datasets with additionally annotated evidences, including [SQuAD](https://www.aclweb.org/anthology/D16-1264/), [CMRC 2018](https://www.aclweb.org/anthology/D19-1600/), RACE<sup>+</sup> (similar to [RACE](https://www.aclweb.org/anthology/D17-1082/)), and [C<sup>3</sup>](https://www.aclweb.org/anthology/2020.tacl-1.10/), covering span-extraction and multiple-choice questions MRC tasks in both English and Chinese. To achieve a higher score in ExpMRC, the model should not only give a correct answer for the question but also give a passage span as the evidence text. We greatly welcome the submission that could be generalized well on different languages and types of MRC tasks with *unsupervised* or *semi-supervised* approaches. **ExpMRC: Explainability Evaluation for Machine Reading Comprehension** - [Yiming Cui](https://ymcui.com), Ting Liu, Wanxiang Che, Zhigang Chen, Shijin Wang - Published in [Heliyon](https://www.cell.com/heliyon) [[Official Publication]](https://www.cell.com/heliyon/fulltext/S2405-8440(22)00578-3) [[arXiv pre-print]](https://arxiv.org/abs/2105.04126) [[**Leaderboard**]](https://ymcui.github.io/expmrc/) [[Papers With Code]](https://paperswithcode.com/dataset/expmrc) ## Submission to Leaderboard Please visit our leaderboard for more information: [https://ymcui.github.io/expmrc/](https://ymcui.github.io/expmrc/) To preserve the integrity of test results and improve the reproducibility, **we do not release the test sets to the public**. Instead, we require you to upload your model onto CodaLab, so that we can run it on the test sets for you. You can follow the instructions on CodaLab (which is similar to SQuAD, CMRC 2018 submission). You can submit your model on one or more subsets in ExpMRC. Sample submission files are shown in `sample_submission` directory. Submission policies: 1. You are free to use any open-source MRC data or automatically generated data for training your systems (both labeled and unlabeled). 2. You are **NOT** allowed to use any **publicly unavailable** human-annotated data for training. 3. We do not encourage using the development set of ExpMRC for training (though it is not prohibited). You should declare whether the system is trained by using the whole/part of the development set. Such submissions will be marked with an asterisk (*). ## Citation If you are using our benchmark in your work, please cite: ``` @article{cui-etal-2022-expmrc, title={ExpMRC: Explainability Evaluation for Machine Reading Comprehension}, author={Cui, Yiming and Liu, Ting and Che, Wanxiang and Chen, Zhigang and Wang, Shijin}, journal={Heliyon}, year={2022}, volume={8}, issue={4}, pages={e09290}, issn={2405-8440}, doi={https://doi.org/10.1016/j.heliyon.2022.e09290} } ``` ## Acknowledgment [Yiming Cui](https://ymcui.com) would like to thank [Google TPU Research Cloud (TRC)](https://g.co/tfrc) program for providing computing resource. We also thank [SQuAD team](https://rajpurkar.github.io/SQuAD-explorer/) for open-sourcing their website template. ## Contact us Please submit an issue.
许可证：CC BY-SA 4.0 任务类别： - 问答 语言： - 中文 - 英文 GitHub 仓库：https://github.com/ymcui/expmrc 随着预训练语言模型（pre-trained language models, PLMs）的发展，在多项机器阅读理解（machine reading comprehension, MRC）数据集上达到人类水平的性能已不再如往昔那般困难。然而，此类模型背后的可解释性依然模糊不清，这引发了学界与工业界对其落地应用的担忧。为提升MRC任务的可解释性，我们提出ExpMRC基准测试集。 **ExpMRC**，即**机器阅读理解可解释性评估（Explainability Evaluation of Machine Reading Comprehension）**，是一款面向机器阅读理解可解释性评估的基准测试集。ExpMRC收录了4个热门MRC数据集的子集，并额外附加了证据标注，涵盖[SQuAD](https://www.aclweb.org/anthology/D16-1264/)、[CMRC 2018](https://www.aclweb.org/anthology/D19-1600/)、RACE<sup>+</sup>（类似[RACE](https://www.aclweb.org/anthology/D17-1082/)）以及[C<sup>3</sup>](https://www.aclweb.org/anthology/2020.tacl-1.10/)，支持英文与中文两种语言，覆盖抽取式问答与多项选择式两类MRC任务。 若想在ExpMRC上获得更高得分，模型不仅需要给出问题的正确答案，还需提供对应的篇章片段作为证据文本。我们诚挚欢迎基于无监督或半监督方法，能够在不同语言与任务类型的MRC场景中实现良好泛化能力的模型提交。 **ExpMRC：机器阅读理解可解释性评估** - [崔一鸣](https://ymcui.com)、刘挺、车万翔、陈智刚、王仕进 - 发表于[Heliyon](https://www.cell.com/heliyon) [[官方出版物]](https://www.cell.com/heliyon/fulltext/S2405-8440(22)00578-3) [[arXiv预印本]](https://arxiv.org/abs/2105.04126) [[**排行榜**]](https://ymcui.github.io/expmrc/) [[Papers With Code]](https://paperswithcode.com/dataset/expmrc) ## 排行榜提交 请访问排行榜页面获取详细信息：[https://ymcui.github.io/expmrc/](https://ymcui.github.io/expmrc/) 为保障测试结果的公正性并提升研究可复现性，**我们不会公开测试集**。取而代之的是，我们要求您将模型上传至CodaLab平台，以便我们在测试集上运行您的模型。您可参考CodaLab平台上的提交指南（流程与SQuAD、CMRC 2018的提交方式类似）。您可选择ExpMRC的一个或多个子集进行模型提交。示例提交文件可在`sample_submission`目录中查看。 提交政策： 1. 您可自由使用任何开源MRC数据集或自动生成的数据（包括带标注与未带标注的数据）训练模型系统。 2. **严禁使用任何未公开的人工标注数据**用于模型训练。 3. 我们不建议使用ExpMRC的开发集进行模型训练（尽管并未明令禁止）。您需声明模型是否使用了全部或部分开发集进行训练，此类提交将被标注星号（*）。 ## 引用 若您的研究工作中使用了本基准测试集，请引用以下文献： @article{cui-etal-2022-expmrc, title={ExpMRC: Explainability Evaluation for Machine Reading Comprehension}, author={Cui, Yiming and Liu, Ting and Che, Wanxiang and Chen, Zhigang and Wang, Shijin}, journal={Heliyon}, year={2022}, volume={8}, issue={4}, pages={e09290}, issn={2405-8440}, doi={https://doi.org/10.1016/j.heliyon.2022.e09290} } ## 致谢 [崔一鸣](https://ymcui.com)感谢Google TPU研究云（TRC）项目提供计算资源支持。我们同时感谢[SQuAD团队](https://rajpurkar.github.io/SQuAD-explorer/)开源其网站模板。 ## 联系我们 请通过提交Issue的方式与我们取得联系。
数据集概述
名称: ExpMRC
目的: 用于评估机器阅读理解（MRC）任务的解释性。
包含内容:
- 四个子集：SQuAD, CMRC 2018, RACE<sup>+</sup>, C<sup>3</sup>
- 覆盖任务类型：span-extraction 和 multiple-choice questions
- 支持语言：中文和英文
使用要求:
- 模型不仅需给出正确答案，还需提供作为证据的文本段落。
- 鼓励使用无监督或半监督方法，并能在不同语言和类型的MRC任务中泛化。
提交规则:
- 可使用任何开源MRC数据或自动生成数据进行训练。
- 禁止使用未公开的人工标注数据进行训练。
- 不鼓励使用ExpMRC的开发集进行训练，如使用需声明。
引用信息:
@article{cui-etal-2022-expmrc, title={ExpMRC: Explainability Evaluation for Machine Reading Comprehension}, author={Cui, Yiming and Liu, Ting and Che, Wanxiang and Chen, Zhigang and Wang, Shijin}, journal={Heliyon}, year={2022}, volume={8}, issue={4}, pages={e09290}, issn={2405-8440}, doi={https://doi.org/10.1016/j.heliyon.2022.e09290} }
