BLiMP

BLiMP是一个包含67个数据集的语言学最小对比基准，用于评估英语语言模型在主要语法现象上的语言知识。每个数据集包含1000对最小对比句，这些句子在语法可接受性和特定语法、形态或语义现象上有所不同。数据集由语言学家设计的语法模板自动生成，人类聚合同意度达到96.4%。BLiMP用于评估n-gram、LSTM和Transformer（GPT-2和TransformerXL）语言模型，通过观察它们是否更倾向于可接受的句子来确定它们对哪些语法差异敏感。

BLiMP is a linguistic minimal contrast benchmark consisting of 67 datasets, designed to evaluate the linguistic knowledge of English language models across major grammatical phenomena. Each dataset contains 1,000 pairs of minimal contrast sentences that differ in grammatical acceptability and specific grammatical, morphological, or semantic phenomena. These datasets are automatically generated from grammatical templates created by linguists, with a human aggregate agreement rate of 96.4%. BLiMP is utilized to evaluate n-gram, LSTM, and Transformer (GPT-2 and TransformerXL) language models, by examining whether they prefer acceptable sentences to determine which grammatical differences they are sensitive to.