lm-diagnostics-negnat
收藏资源简介：
LM Diagnostics (cprag)是一个用于对语言模型进行心理语言学诊断的数据集，由Allyson Ettinger在论文“What BERT Is Not: Lessons from a new suite of psycholinguistic diagnostics for language models”中引入。该数据集通过一系列来自人类语言实验的诊断，允许我们针对语言模型在生成上下文预测时使用的信息提出有针对性的问题。作为一个案例研究，这些诊断被应用于流行的BERT模型，发现它能够区分涉及共享类别或角色反转的好与坏的完成情况，但敏感性不如人类，并且它能够稳健地检索名词上位词，但在处理具有挑战性的推理和基于角色的事件预测方面存在困难，特别是对否定语境的影响表现出明显的敏感性不足。
LM Diagnostics (cprag) is a psycholinguistic diagnosis dataset for language models, introduced by Allyson Ettinger in the paper "What BERT Is Not: Lessons from a new suite of psycholinguistic diagnostics for language models". This dataset leverages a series of diagnostics derived from human language experiments to enable targeted inquiries into the information that language models utilize when generating contextual predictions. As a case study, these diagnostics were applied to the prevalent BERT model, and the findings reveal that the model can distinguish between well-formed and ill-formed completions involving shared categories or role reversals, yet its sensitivity is inferior to that of humans. It can robustly retrieve noun hypernyms, but struggles with challenging reasoning and role-based event prediction tasks, particularly exhibiting a marked insensitivity to the impacts of negation contexts.
数据集概述
基本信息
- 名称: LM Diagnostics (cprag) Clone
- 许可证: MIT License
- 语言: 英语 (en)
- 数据规模: 小于1K样本 (n<1K)
数据集描述
该数据集是一个诊断性数据集 (cprag)，源自论文《What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models》（作者：Allyson Ettinger）。该论文通过一系列心理语言学诊断测试，探究语言模型（如BERT）在预训练过程中所获得的具体语言能力，重点关注模型在上下文预测中使用的信息。
关键研究内容
- 评估模型区分涉及共享类别或角色反转的合理与不合理补全的能力。
- 测试模型在名词上位词检索方面的表现。
- 分析模型在复杂推理和基于角色的事件预测中的挑战。
- 特别关注模型对否定语境影响的敏感性。
引用信息
bibtex @article{10.1162/tacl_a_00298, author = {Ettinger, Allyson}, title = {What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models}, journal = {Transactions of the Association for Computational Linguistics}, volume = {8}, pages = {34-48}, year = {2020}, month = {01}, abstract = {Pre-training by language modeling has become a popular and successful approach to NLP tasks, but we have yet to understand exactly what linguistic capacities these pre-training processes confer upon models. In this paper we introduce a suite of diagnostics drawn from human language experiments, which allow us to ask targeted questions about information used by language models for generating predictions in context. As a case study, we apply these diagnostics to the popular BERT model, finding that it can generally distinguish good from bad completions involving shared category or role reversal, albeit with less sensitivity than humans, and it robustly retrieves noun hypernyms, but it struggles with challenging inference and role-based event prediction— and, in particular, it shows clear insensitivity to the contextual impacts of negation.}, issn = {2307-387X}, doi = {10.1162/tacl_a_00298}, url = {https://doi.org/10.1162/tacl_a_00298}, eprint = {https://direct.mit.edu/tacl/article-pdf/doi/10.1162/tacl_a_00298/1923116/tacl_a_00298.pdf}, }
