blimp
收藏魔搭社区2025-11-27 更新2025-09-20 收录
下载链接：
https://modelscope.cn/datasets/nyu-mll/blimp
下载链接
链接失效反馈官方服务：
资源简介：
# Dataset Card for "blimp"
## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)
## Dataset Description
- **Homepage:**
- **Repository:** https://github.com/alexwarstadt/blimp
- **Paper:** [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://doi.org/10.1162/tacl_a_00321)
- **Paper:** https://arxiv.org/abs/1912.00582
- **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **Size of downloaded dataset files:** 29.58 MB
- **Size of the generated dataset:** 11.45 MB
- **Total amount of disk used:** 41.03 MB
### Dataset Summary
BLiMP is a challenge set for evaluating what language models (LMs) know about
major grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each
containing 1000 minimal pairs isolating specific contrasts in syntax,
morphology, or semantics. The data is automatically generated according to
expert-crafted grammars.
### Supported Tasks and Leaderboards
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Languages
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## Dataset Structure
### Data Instances
#### adjunct_island
- **Size of downloaded dataset files:** 0.36 MB
- **Size of the generated dataset:** 0.17 MB
- **Total amount of disk used:** 0.52 MB
An example of 'train' looks as follows.
```
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
```
#### anaphor_gender_agreement
- **Size of downloaded dataset files:** 0.44 MB
- **Size of the generated dataset:** 0.14 MB
- **Total amount of disk used:** 0.57 MB
An example of 'train' looks as follows.
```
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
```
#### anaphor_number_agreement
- **Size of downloaded dataset files:** 0.45 MB
- **Size of the generated dataset:** 0.14 MB
- **Total amount of disk used:** 0.59 MB
An example of 'train' looks as follows.
```
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
```
#### animate_subject_passive
- **Size of downloaded dataset files:** 0.46 MB
- **Size of the generated dataset:** 0.15 MB
- **Total amount of disk used:** 0.61 MB
An example of 'train' looks as follows.
```
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
```
#### animate_subject_trans
- **Size of downloaded dataset files:** 0.43 MB
- **Size of the generated dataset:** 0.13 MB
- **Total amount of disk used:** 0.57 MB
An example of 'train' looks as follows.
```
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
```
### Data Fields
The data fields are the same among all splits.
#### adjunct_island
- `sentence_good`: a `string` feature.
- `sentence_bad`: a `string` feature.
- `field`: a `string` feature.
- `linguistics_term`: a `string` feature.
- `UID`: a `string` feature.
- `simple_LM_method`: a `bool` feature.
- `one_prefix_method`: a `bool` feature.
- `two_prefix_method`: a `bool` feature.
- `lexically_identical`: a `bool` feature.
- `pair_id`: a `int32` feature.
#### anaphor_gender_agreement
- `sentence_good`: a `string` feature.
- `sentence_bad`: a `string` feature.
- `field`: a `string` feature.
- `linguistics_term`: a `string` feature.
- `UID`: a `string` feature.
- `simple_LM_method`: a `bool` feature.
- `one_prefix_method`: a `bool` feature.
- `two_prefix_method`: a `bool` feature.
- `lexically_identical`: a `bool` feature.
- `pair_id`: a `int32` feature.
#### anaphor_number_agreement
- `sentence_good`: a `string` feature.
- `sentence_bad`: a `string` feature.
- `field`: a `string` feature.
- `linguistics_term`: a `string` feature.
- `UID`: a `string` feature.
- `simple_LM_method`: a `bool` feature.
- `one_prefix_method`: a `bool` feature.
- `two_prefix_method`: a `bool` feature.
- `lexically_identical`: a `bool` feature.
- `pair_id`: a `int32` feature.
#### animate_subject_passive
- `sentence_good`: a `string` feature.
- `sentence_bad`: a `string` feature.
- `field`: a `string` feature.
- `linguistics_term`: a `string` feature.
- `UID`: a `string` feature.
- `simple_LM_method`: a `bool` feature.
- `one_prefix_method`: a `bool` feature.
- `two_prefix_method`: a `bool` feature.
- `lexically_identical`: a `bool` feature.
- `pair_id`: a `int32` feature.
#### animate_subject_trans
- `sentence_good`: a `string` feature.
- `sentence_bad`: a `string` feature.
- `field`: a `string` feature.
- `linguistics_term`: a `string` feature.
- `UID`: a `string` feature.
- `simple_LM_method`: a `bool` feature.
- `one_prefix_method`: a `bool` feature.
- `two_prefix_method`: a `bool` feature.
- `lexically_identical`: a `bool` feature.
- `pair_id`: a `int32` feature.
### Data Splits
| name |train|
|------------------------|----:|
|adjunct_island | 1000|
|anaphor_gender_agreement| 1000|
|anaphor_number_agreement| 1000|
|animate_subject_passive | 1000|
|animate_subject_trans | 1000|
## Dataset Creation
### Curation Rationale
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Source Data
#### Initial Data Collection and Normalization
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
#### Who are the source language producers?
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Annotations
#### Annotation process
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
#### Who are the annotators?
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Personal and Sensitive Information
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## Considerations for Using the Data
### Social Impact of Dataset
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Discussion of Biases
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Other Known Limitations
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## Additional Information
### Dataset Curators
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Licensing Information
BLiMP is distributed under a [CC-BY](https://creativecommons.org/licenses/by/4.0/) license. Source: https://github.com/alexwarstadt/blimp#license
### Citation Information
```
@article{warstadt2020blimp,
author = {Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R.},
title = {BLiMP: The Benchmark of Linguistic Minimal Pairs for English},
journal = {Transactions of the Association for Computational Linguistics},
volume = {8},
number = {},
pages = {377-392},
year = {2020},
doi = {10.1162/tacl\_a\_00321},
URL = {https://doi.org/10.1162/tacl_a_00321},
eprint = {https://doi.org/10.1162/tacl_a_00321},
abstract = { We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4\%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands. }
}
```
#### Errata
Some results were misreported in the published TACL version. Please refer to the corrected version on arXiv: https://arxiv.org/abs/1912.00582
### Contributions
Thanks to [@lhoestq](https://github.com/lhoestq), [@patrickvonplaten](https://github.com/patrickvonplaten), [@thomwolf](https://github.com/thomwolf) for adding this dataset.
# "BLiMP"数据集卡片
## 目录
- [数据集描述](#dataset-description)
- [数据集摘要](#dataset-summary)
- [支持任务与排行榜](#supported-tasks-and-leaderboards)
- [语言](#languages)
- [数据集结构](#dataset-structure)
- [数据实例](#data-instances)
- [数据字段](#data-fields)
- [数据拆分](#data-splits)
- [数据集构建](#dataset-creation)
- [构建初衷](#curation-rationale)
- [源数据](#source-data)
- [标注](#annotations)
- [个人与敏感信息](#personal-and-sensitive-information)
- [数据集使用注意事项](#considerations-for-using-the-data)
- [数据集社会影响](#social-impact-of-dataset)
- [偏差讨论](#discussion-of-biases)
- [其他已知局限性](#other-known-limitations)
- [附加信息](#additional-information)
- [数据集维护者](#dataset-curators)
- [许可信息](#licensing-information)
- [引用信息](#citation-information)
- [贡献者](#contributions)
## 数据集描述
- **主页：**
- **代码仓库：** https://github.com/alexwarstadt/blimp
- **论文：** [BLiMP：面向英语的语言学极小对基准测试集](https://doi.org/10.1162/tacl_a_00321)
- **论文：** https://arxiv.org/abs/1912.00582
- **联络人：** [更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
- **下载数据集文件大小：** 29.58 MB
- **生成数据集大小：** 11.45 MB
- **总磁盘占用：** 41.03 MB
### 数据集摘要
BLiMP是一款用于评估语言模型（Language Model, LM）对英语主要语法现象掌握程度的挑战集。该数据集包含67个子集，每个子集包含1000组极小对——即成对的、差异极小的句子，这些句子在语法可接受性上形成对比，并隔离句法、词法或语义层面的特定现象。数据集依据语言学家精心设计的语法模板自动生成。
### 支持任务与排行榜
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 语言
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## 数据集结构
### 数据实例
#### 附加语孤岛（adjunct_island）
- **下载数据集文件大小：** 0.36 MB
- **生成数据集大小：** 0.17 MB
- **总磁盘占用：** 0.52 MB
训练集的一条示例如下所示：
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
#### 照应语性别一致性（anaphor_gender_agreement）
- **下载数据集文件大小：** 0.44 MB
- **生成数据集大小：** 0.14 MB
- **总磁盘占用：** 0.57 MB
训练集的一条示例如下所示：
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
#### 照应语数一致性（anaphor_number_agreement）
- **下载数据集文件大小：** 0.45 MB
- **生成数据集大小：** 0.14 MB
- **总磁盘占用：** 0.59 MB
训练集的一条示例如下所示：
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
#### 有生命主语被动句（animate_subject_passive）
- **下载数据集文件大小：** 0.46 MB
- **生成数据集大小：** 0.15 MB
- **总磁盘占用：** 0.61 MB
训练集的一条示例如下所示：
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
#### 有生命主语及物句（animate_subject_trans）
- **下载数据集文件大小：** 0.43 MB
- **生成数据集大小：** 0.13 MB
- **总磁盘占用：** 0.57 MB
训练集的一条示例如下所示：
{
"UID": "tough_vs_raising_1",
"field": "syntax_semantics",
"lexically_identical": false,
"linguistics_term": "control_raising",
"one_prefix_method": false,
"pair_id": 2,
"sentence_bad": "Benjamin's tutor was certain to boast about.",
"sentence_good": "Benjamin's tutor was easy to boast about.",
"simple_LM_method": true,
"two_prefix_method": false
}
### 数据字段
所有数据拆分下的字段均保持统一。
#### 附加语孤岛（adjunct_island）
- `sentence_good`: 字符串类型特征
- `sentence_bad`: 字符串类型特征
- `field`: 字符串类型特征
- `linguistics_term`: 字符串类型特征
- `UID`: 字符串类型特征
- `simple_LM_method`: 布尔类型特征
- `one_prefix_method`: 布尔类型特征
- `two_prefix_method`: 布尔类型特征
- `lexically_identical`: 布尔类型特征
- `pair_id`: int32类型特征
#### 照应语性别一致性（anaphor_gender_agreement）
- `sentence_good`: 字符串类型特征
- `sentence_bad`: 字符串类型特征
- `field`: 字符串类型特征
- `linguistics_term`: 字符串类型特征
- `UID`: 字符串类型特征
- `simple_LM_method`: 布尔类型特征
- `one_prefix_method`: 布尔类型特征
- `two_prefix_method`: 布尔类型特征
- `lexically_identical`: 布尔类型特征
- `pair_id`: int32类型特征
#### 照应语数一致性（anaphor_number_agreement）
- `sentence_good`: 字符串类型特征
- `sentence_bad`: 字符串类型特征
- `field`: 字符串类型特征
- `linguistics_term`: 字符串类型特征
- `UID`: 字符串类型特征
- `simple_LM_method`: 布尔类型特征
- `one_prefix_method`: 布尔类型特征
- `two_prefix_method`: 布尔类型特征
- `lexically_identical`: 布尔类型特征
- `pair_id`: int32类型特征
#### 有生命主语被动句（animate_subject_passive）
- `sentence_good`: 字符串类型特征
- `sentence_bad`: 字符串类型特征
- `field`: 字符串类型特征
- `linguistics_term`: 字符串类型特征
- `UID`: 字符串类型特征
- `simple_LM_method`: 布尔类型特征
- `one_prefix_method`: 布尔类型特征
- `two_prefix_method`: 布尔类型特征
- `lexically_identical`: 布尔类型特征
- `pair_id`: int32类型特征
#### 有生命主语及物句（animate_subject_trans）
- `sentence_good`: 字符串类型特征
- `sentence_bad`: 字符串类型特征
- `field`: 字符串类型特征
- `linguistics_term`: 字符串类型特征
- `UID`: 字符串类型特征
- `simple_LM_method`: 布尔类型特征
- `one_prefix_method`: 布尔类型特征
- `two_prefix_method`: 布尔类型特征
- `lexically_identical`: 布尔类型特征
- `pair_id`: int32类型特征
### 数据拆分
| 数据集名称 | 训练集样本数 |
|---------------------------|-------------:|
| 附加语孤岛 | 1000 |
| 照应语性别一致性 | 1000 |
| 照应语数一致性 | 1000 |
| 有生命主语被动句 | 1000 |
| 有生命主语及物句 | 1000 |
## 数据集构建
### 构建初衷
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 源数据
#### 初始数据收集与归一化
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
#### 源语言生产者是谁？
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 标注
#### 标注流程
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
#### 标注者是谁？
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 个人与敏感信息
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## 数据集使用注意事项
### 数据集社会影响
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 偏差讨论
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 其他已知局限性
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## 附加信息
### 数据集维护者
[更多信息待补充](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### 许可信息
BLiMP采用[CC-BY](https://creativecommons.org/licenses/by/4.0/)许可协议发布。来源：https://github.com/alexwarstadt/blimp#license
### 引用信息
@article{warstadt2020blimp,
author = {Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R.},
title = {BLiMP: The Benchmark of Linguistic Minimal Pairs for English},
journal = {Transactions of the Association for Computational Linguistics},
volume = {8},
number = {},
pages = {377-392},
year = {2020},
doi = {10.1162/tacl_a_00321},
URL = {https://doi.org/10.1162/tacl_a_00321},
eprint = {https://doi.org/10.1162/tacl_a_00321},
abstract = { 我们推出了语言学极小对基准测试集（BLiMP），这是一款用于评估语言模型（Language Model, LM）英语语法知识的挑战集。BLiMP包含67个独立子集，每个子集包含1000组极小对——即成对的、差异极小的句子，这些句子在语法可接受性上形成对比，并隔离句法、词法或语义层面的特定现象。数据集依据语言学家精心设计的语法模板生成，人类标注者对标签的总体一致性达96.4%。我们通过观察n-gram、LSTM以及Transformer（GPT-2与Transformer-XL）模型是否为每组极小对中的合格句分配更高概率，来评估这些模型的表现。实验发现，当前主流模型能够可靠识别与一致性相关的词法对比，但在处理一些微妙的语义与句法现象时表现欠佳，例如否定极项与提取孤岛。 }
}
#### 勘误
已发表的TACL版本中存在部分结果报告错误，请参阅arXiv上的修正版本：https://arxiv.org/abs/1912.00582
### 贡献者
感谢[@lhoestq](https://github.com/lhoestq)、[@patrickvonplaten](https://github.com/patrickvonplaten)、[@thomwolf](https://github.com/thomwolf)为本数据集的收录提供支持。
提供机构：
maas
创建时间：
2025-09-17
