quincyqiang/test

--- annotations_creators: - other language_creators: - other language: - en license: - cc-by-4.0 multilinguality: - monolingual size_categories: - 10K<n<100K source_datasets: - original task_categories: - text-classification task_ids: - acceptability-classification - natural-language-inference - semantic-similarity-scoring - sentiment-classification - text-scoring paperswithcode_id: glue pretty_name: GLUE (General Language Understanding Evaluation benchmark) train-eval-index: - config: cola task: text-classification task_id: binary_classification splits: train_split: train eval_split: validation col_mapping: sentence: text label: target - config: sst2 task: text-classification task_id: binary_classification splits: train_split: train eval_split: validation col_mapping: sentence: text label: target - config: mrpc task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: sentence1: text1 sentence2: text2 label: target - config: qqp task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: question1: text1 question2: text2 label: target - config: stsb task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: sentence1: text1 sentence2: text2 label: target - config: mnli task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation_matched col_mapping: premise: text1 hypothesis: text2 label: target - config: mnli_mismatched task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: premise: text1 hypothesis: text2 label: target - config: mnli_matched task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: premise: text1 hypothesis: text2 label: target - config: qnli task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: question: text1 sentence: text2 label: target - config: rte task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: sentence1: text1 sentence2: text2 label: target - config: wnli task: text-classification task_id: natural_language_inference splits: train_split: train eval_split: validation col_mapping: sentence1: text1 sentence2: text2 label: target configs: - ax - cola - mnli - mnli_matched - mnli_mismatched - mrpc - qnli - qqp - rte - sst2 - stsb - wnli tags: - qa-nli - coreference-nli - paraphrase-identification --- # Dataset Card for GLUE ## Table of Contents - [Dataset Card for GLUE](#dataset-card-for-glue) - [Table of Contents](#table-of-contents) - [Dataset Description](#dataset-description) - [Dataset Summary](#dataset-summary) - [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards) - [ax](#ax) - [cola](#cola) - [mnli](#mnli) - [mnli_matched](#mnli_matched) - [mnli_mismatched](#mnli_mismatched) - [mrpc](#mrpc) - [qnli](#qnli) - [qqp](#qqp) - [rte](#rte) - [sst2](#sst2) - [stsb](#stsb) - [wnli](#wnli) - [Languages](#languages) - [Dataset Structure](#dataset-structure) - [Data Instances](#data-instances) - [ax](#ax-1) - [cola](#cola-1) - [mnli](#mnli-1) - [mnli_matched](#mnli_matched-1) - [mnli_mismatched](#mnli_mismatched-1) - [mrpc](#mrpc-1) - [qnli](#qnli-1) - [qqp](#qqp-1) - [rte](#rte-1) - [sst2](#sst2-1) - [stsb](#stsb-1) - [wnli](#wnli-1) - [Data Fields](#data-fields) - [ax](#ax-2) - [cola](#cola-2) - [mnli](#mnli-2) - [mnli_matched](#mnli_matched-2) - [mnli_mismatched](#mnli_mismatched-2) - [mrpc](#mrpc-2) - [qnli](#qnli-2) - [qqp](#qqp-2) - [rte](#rte-2) - [sst2](#sst2-2) - [stsb](#stsb-2) - [wnli](#wnli-2) - [Data Splits](#data-splits) - [ax](#ax-3) - [cola](#cola-3) - [mnli](#mnli-3) - [mnli_matched](#mnli_matched-3) - [mnli_mismatched](#mnli_mismatched-3) - [mrpc](#mrpc-3) - [qnli](#qnli-3) - [qqp](#qqp-3) - [rte](#rte-3) - [sst2](#sst2-3) - [stsb](#stsb-3) - [wnli](#wnli-3) - [Dataset Creation](#dataset-creation) - [Curation Rationale](#curation-rationale) - [Source Data](#source-data) - [Initial Data Collection and Normalization](#initial-data-collection-and-normalization) - [Who are the source language producers?](#who-are-the-source-language-producers) - [Annotations](#annotations) - [Annotation process](#annotation-process) - [Who are the annotators?](#who-are-the-annotators) - [Personal and Sensitive Information](#personal-and-sensitive-information) - [Considerations for Using the Data](#considerations-for-using-the-data) - [Social Impact of Dataset](#social-impact-of-dataset) - [Discussion of Biases](#discussion-of-biases) - [Other Known Limitations](#other-known-limitations) - [Additional Information](#additional-information) - [Dataset Curators](#dataset-curators) - [Licensing Information](#licensing-information) - [Citation Information](#citation-information) - [Contributions](#contributions) ## Dataset Description - **Homepage:** [https://nyu-mll.github.io/CoLA/](https://nyu-mll.github.io/CoLA/) - **Repository:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) - **Paper:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) - **Point of Contact:** [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) - **Size of downloaded dataset files:** 955.33 MB - **Size of the generated dataset:** 229.68 MB - **Total amount of disk used:** 1185.01 MB ### Dataset Summary GLUE, the General Language Understanding Evaluation benchmark (https://gluebenchmark.com/) is a collection of resources for training, evaluating, and analyzing natural language understanding systems. ### Supported Tasks and Leaderboards The leaderboard for the GLUE benchmark can be found [at this address](https://gluebenchmark.com/). It comprises the following tasks: #### ax A manually-curated evaluation dataset for fine-grained analysis of system performance on a broad range of linguistic phenomena. This dataset evaluates sentence understanding through Natural Language Inference (NLI) problems. Use a model trained on MulitNLI to produce predictions for this dataset. #### cola The Corpus of Linguistic Acceptability consists of English acceptability judgments drawn from books and journal articles on linguistic theory. Each example is a sequence of words annotated with whether it is a grammatical English sentence. #### mnli The Multi-Genre Natural Language Inference Corpus is a crowdsourced collection of sentence pairs with textual entailment annotations. Given a premise sentence and a hypothesis sentence, the task is to predict whether the premise entails the hypothesis (entailment), contradicts the hypothesis (contradiction), or neither (neutral). The premise sentences are gathered from ten different sources, including transcribed speech, fiction, and government reports. The authors of the benchmark use the standard test set, for which they obtained private labels from the RTE authors, and evaluate on both the matched (in-domain) and mismatched (cross-domain) section. They also uses and recommend the SNLI corpus as 550k examples of auxiliary training data. #### mnli_matched The matched validation and test splits from MNLI. See the "mnli" BuilderConfig for additional information. #### mnli_mismatched The mismatched validation and test splits from MNLI. See the "mnli" BuilderConfig for additional information. #### mrpc The Microsoft Research Paraphrase Corpus (Dolan & Brockett, 2005) is a corpus of sentence pairs automatically extracted from online news sources, with human annotations for whether the sentences in the pair are semantically equivalent. #### qnli The Stanford Question Answering Dataset is a question-answering dataset consisting of question-paragraph pairs, where one of the sentences in the paragraph (drawn from Wikipedia) contains the answer to the corresponding question (written by an annotator). The authors of the benchmark convert the task into sentence pair classification by forming a pair between each question and each sentence in the corresponding context, and filtering out pairs with low lexical overlap between the question and the context sentence. The task is to determine whether the context sentence contains the answer to the question. This modified version of the original task removes the requirement that the model select the exact answer, but also removes the simplifying assumptions that the answer is always present in the input and that lexical overlap is a reliable cue. #### qqp The Quora Question Pairs2 dataset is a collection of question pairs from the community question-answering website Quora. The task is to determine whether a pair of questions are semantically equivalent. #### rte The Recognizing Textual Entailment (RTE) datasets come from a series of annual textual entailment challenges. The authors of the benchmark combined the data from RTE1 (Dagan et al., 2006), RTE2 (Bar Haim et al., 2006), RTE3 (Giampiccolo et al., 2007), and RTE5 (Bentivogli et al., 2009). Examples are constructed based on news and Wikipedia text. The authors of the benchmark convert all datasets to a two-class split, where for three-class datasets they collapse neutral and contradiction into not entailment, for consistency. #### sst2 The Stanford Sentiment Treebank consists of sentences from movie reviews and human annotations of their sentiment. The task is to predict the sentiment of a given sentence. It uses the two-way (positive/negative) class split, with only sentence-level labels. #### stsb The Semantic Textual Similarity Benchmark (Cer et al., 2017) is a collection of sentence pairs drawn from news headlines, video and image captions, and natural language inference data. Each pair is human-annotated with a similarity score from 1 to 5. #### wnli The Winograd Schema Challenge (Levesque et al., 2011) is a reading comprehension task in which a system must read a sentence with a pronoun and select the referent of that pronoun from a list of choices. The examples are manually constructed to foil simple statistical methods: Each one is contingent on contextual information provided by a single word or phrase in the sentence. To convert the problem into sentence pair classification, the authors of the benchmark construct sentence pairs by replacing the ambiguous pronoun with each possible referent. The task is to predict if the sentence with the pronoun substituted is entailed by the original sentence. They use a small evaluation set consisting of new examples derived from fiction books that was shared privately by the authors of the original corpus. While the included training set is balanced between two classes, the test set is imbalanced between them (65% not entailment). Also, due to a data quirk, the development set is adversarial: hypotheses are sometimes shared between training and development examples, so if a model memorizes the training examples, they will predict the wrong label on corresponding development set example. As with QNLI, each example is evaluated separately, so there is not a systematic correspondence between a model's score on this task and its score on the unconverted original task. The authors of the benchmark call converted dataset WNLI (Winograd NLI). ### Languages The language data in GLUE is in English (BCP-47 `en`) ## Dataset Structure ### Data Instances #### ax - **Size of downloaded dataset files:** 0.21 MB - **Size of the generated dataset:** 0.23 MB - **Total amount of disk used:** 0.44 MB An example of 'test' looks as follows. ``` { "premise": "The cat sat on the mat.", "hypothesis": "The cat did not sit on the mat.", "label": -1, "idx: 0 } ``` #### cola - **Size of downloaded dataset files:** 0.36 MB - **Size of the generated dataset:** 0.58 MB - **Total amount of disk used:** 0.94 MB An example of 'train' looks as follows. ``` { "sentence": "Our friends won't buy this analysis, let alone the next one we propose.", "label": 1, "id": 0 } ``` #### mnli - **Size of downloaded dataset files:** 298.29 MB - **Size of the generated dataset:** 78.65 MB - **Total amount of disk used:** 376.95 MB An example of 'train' looks as follows. ``` { "premise": "Conceptually cream skimming has two basic dimensions - product and geography.", "hypothesis": "Product and geography are what make cream skimming work.", "label": 1, "idx": 0 } ``` #### mnli_matched - **Size of downloaded dataset files:** 298.29 MB - **Size of the generated dataset:** 3.52 MB - **Total amount of disk used:** 301.82 MB An example of 'test' looks as follows. ``` { "premise": "Hierbas, ans seco, ans dulce, and frigola are just a few names worth keeping a look-out for.", "hypothesis": "Hierbas is a name worth looking out for.", "label": -1, "idx": 0 } ``` #### mnli_mismatched - **Size of downloaded dataset files:** 298.29 MB - **Size of the generated dataset:** 3.73 MB - **Total amount of disk used:** 302.02 MB An example of 'test' looks as follows. ``` { "premise": "What have you decided, what are you going to do?", "hypothesis": "So what's your decision?, "label": -1, "idx": 0 } ``` #### mrpc [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### qnli [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### qqp [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### rte [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### sst2 [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### stsb [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### wnli [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Data Fields The data fields are the same among all splits. #### ax - `premise`: a `string` feature. - `hypothesis`: a `string` feature. - `label`: a classification label, with possible values including `entailment` (0), `neutral` (1), `contradiction` (2). - `idx`: a `int32` feature. #### cola - `sentence`: a `string` feature. - `label`: a classification label, with possible values including `unacceptable` (0), `acceptable` (1). - `idx`: a `int32` feature. #### mnli - `premise`: a `string` feature. - `hypothesis`: a `string` feature. - `label`: a classification label, with possible values including `entailment` (0), `neutral` (1), `contradiction` (2). - `idx`: a `int32` feature. #### mnli_matched - `premise`: a `string` feature. - `hypothesis`: a `string` feature. - `label`: a classification label, with possible values including `entailment` (0), `neutral` (1), `contradiction` (2). - `idx`: a `int32` feature. #### mnli_mismatched - `premise`: a `string` feature. - `hypothesis`: a `string` feature. - `label`: a classification label, with possible values including `entailment` (0), `neutral` (1), `contradiction` (2). - `idx`: a `int32` feature. #### mrpc [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### qnli [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### qqp [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### rte [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### sst2 [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### stsb [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### wnli [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Data Splits #### ax | |test| |---|---:| |ax |1104| #### cola | |train|validation|test| |----|----:|---------:|---:| |cola| 8551| 1043|1063| #### mnli | |train |validation_matched|validation_mismatched|test_matched|test_mismatched| |----|-----:|-----------------:|--------------------:|-----------:|--------------:| |mnli|392702| 9815| 9832| 9796| 9847| #### mnli_matched | |validation|test| |------------|---------:|---:| |mnli_matched| 9815|9796| #### mnli_mismatched | |validation|test| |---------------|---------:|---:| |mnli_mismatched| 9832|9847| #### mrpc [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### qnli [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### qqp [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### rte [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### sst2 [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### stsb [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### wnli [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ## Dataset Creation ### Curation Rationale [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Source Data #### Initial Data Collection and Normalization [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### Who are the source language producers? [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Annotations #### Annotation process [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) #### Who are the annotators? [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Personal and Sensitive Information [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ## Considerations for Using the Data ### Social Impact of Dataset [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Discussion of Biases [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Other Known Limitations [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ## Additional Information ### Dataset Curators [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Licensing Information [More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards) ### Citation Information ``` @article{warstadt2018neural, title={Neural Network Acceptability Judgments}, author={Warstadt, Alex and Singh, Amanpreet and Bowman, Samuel R}, journal={arXiv preprint arXiv:1805.12471}, year={2018} } @inproceedings{wang2019glue, title={{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding}, author={Wang, Alex and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel R.}, note={In the Proceedings of ICLR.}, year={2019} } Note that each GLUE dataset has its own citation. Please see the source to see the correct citation for each contained dataset. ``` ### Contributions Thanks to [@patpizio](https://github.com/patpizio), [@jeswan](https://github.com/jeswan), [@thomwolf](https://github.com/thomwolf), [@patrickvonplaten](https://github.com/patrickvonplaten), [@mariamabarham](https://github.com/mariamabarham) for adding this dataset.

原始信息汇总

数据集概述

数据集基本信息

• 名称: GLUE (General Language Understanding Evaluation benchmark)
• 语言: 英语 (en)
• 许可证: CC-BY-4.0
• 多语言性: 单语种
• 大小: 10K<n<100K
• 源数据集: 原始数据
• 任务类别: 文本分类
• 任务ID:
  • acceptability-classification
  • natural-language-inference
  • semantic-similarity-scoring
  • sentiment-classification
  • text-scoring
• 配置:
  • ax
  • cola
  • mnli
  • mnli_matched
  • mnli_mismatched
  • mrpc
  • qnli
  • qqp
  • rte
  • sst2
  • stsb
  • wnli

数据集结构

数据实例

• ax:

  • 下载大小: 0.21 MB
  • 生成大小: 0.23 MB
  • 总磁盘使用: 0.44 MB
  • 示例字段: premise, hypothesis, label, idx

• cola:

  • 下载大小: 0.36 MB
  • 生成大小: 0.58 MB
  • 总磁盘使用: 0.94 MB
  • 示例字段: sentence, label, id

• mnli:

  • 下载大小: 298.29 MB
  • 生成大小: 78.65 MB
  • 总磁盘使用: 376.95 MB
  • 示例字段: premise, hypothesis, label, idx

• mnli_matched:

  • 下载大小: 298.29 MB
  • 生成大小: 3.52 MB
  • 总磁盘使用: 301.82 MB
  • 示例字段: premise, hypothesis, label, idx

• mnli_mismatched:

  • 下载大小: 298.29 MB
  • 生成大小: 3.73 MB
  • 总磁盘使用: 302.02 MB
  • 示例字段: premise, hypothesis, label, idx

数据字段

• ax:

  • premise: 字符串
  • hypothesis: 字符串
  • label: 分类标签 (entailment, neutral, contradiction)
  • idx: 整数

• cola:

  • sentence: 字符串
  • label: 分类标签 (unacceptable, acceptable)
  • idx: 整数

• mnli:

  • premise: 字符串
  • hypothesis: 字符串
  • label: 分类标签 (entailment, neutral, contradiction)
  • idx: 整数

• mnli_matched:

  • premise: 字符串
  • hypothesis: 字符串
  • label: 分类标签 (entailment, neutral, contradiction)
  • idx: 整数

• mnli_mismatched:

  • premise: 字符串
  • hypothesis: 字符串
  • label: 分类标签 (entailment, neutral, contradiction)
  • idx: 整数

数据分割

• ax:

  • test: 1104

• cola:

  • train: 8551
  • validation: 1043
  • test: 1063

• mnli:

  • train: 392702
  • validation_matched: 9815
  • validation_mismatched: 9832
  • test_matched: 9796
  • test_mismatched: 9847

• mnli_matched:

  • validation: 9815
  • test: 9796

• mnli_mismatched:

  • validation: 9832
  • test: 9847

搜集汇总

数据集介绍

构建方式

GLUE基准测试集由多个源自不同研究领域的自然语言理解子数据集有机融合而成，旨在为模型的训练、评估与分析提供统一平台。其构建方式体现了高度的系统性与综合性：从语言学可接受性判断（CoLA）、情感分析（SST-2）、语义等价性判别（MRPC、QQP）到自然语言推理（MNLI、RTE、WNLI）、问答推理（QNLI）及语义文本相似度评分（STS-B）等任务中，精选经典或经过适配改造的数据资源。各子集均保留了原始标注，部分任务如QNLI通过将问答对转化为句子对分类问题进行重构，而RTE则将三分类任务统一为二分类格式，确保了基准测试的标准化与可比性。

特点

该数据集最显著的特征在于其多任务性与评估维度的丰富性。它囊括了文本分类、自然语言推理、语义相似度计算及可接受性判断等多种核心NLU任务，能够全面衡量模型在不同语言理解层面的表现。数据规模介于1万至10万之间，涵盖从单句分类到句子对关系推断的多样输入格式。此外，MNLI子集特别设计了匹配（in-domain）与不匹配（cross-domain）两种验证/测试划分，用于评估模型的领域泛化能力。WNLI子集则因其对抗性样本设计而独具挑战性，能够有效检验模型对细粒度语言现象的敏感度与鲁棒性。

使用方法

GLUE数据集可通过Hugging Face的datasets库便捷加载，用户只需指定子集配置名称（如'cola'、'sst2'）即可获取对应数据。每个子集均预定义了训练、验证与测试的划分，并提供了清晰的字段映射关系，如将原始字段统一映射为'text'、'text1'、'text2'及'target'，便于直接用于主流深度学习框架的模型训练。对于MNLI子集，用户需注意其包含'matched'与'mismatched'两种验证集，以分别评估同领域与跨领域性能。评估时，建议参考官方排行榜的指标定义，针对不同任务选用准确率、F1分数或皮尔逊相关系数等恰当度量标准。

背景与挑战

背景概述

GLUE（General Language Understanding Evaluation）基准测试集由纽约大学、华盛顿大学等多家顶尖机构的研究者于2018年联合创建，旨在系统评估自然语言理解模型的综合能力。该数据集整合了涵盖语法可接受性、情感分析、文本蕴含、语义相似度及问答推理等九项经典自然语言处理任务，包括CoLA、SST-2、MNLI、QQP等子集，为预训练语言模型（如BERT、RoBERTa）的性能评测提供了标准化平台。GLUE的提出解决了此前自然语言理解领域缺乏统一评估框架的困境，其影响力迅速扩展至整个学术界与工业界，成为衡量模型语言理解能力的核心标杆，推动了诸如多任务学习、跨任务泛化等研究方向的发展。

当前挑战

GLUE面临的核心挑战在于如何真实反映模型对自然语言的深层理解能力。首先，各子任务之间存在显著的领域差异与难度不均，例如WNLI的指代消解任务因数据集规模小且存在对抗性样本，导致模型难以泛化；而MNLI的跨领域测试则要求模型具备强大的鲁棒性。其次，数据构建过程中引入了固有偏差，如SST-2的情感标签分布不均、QQP中语义等价判断的主观性，这些因素可能使模型依赖浅层统计特征而非真正理解语义。此外，GLUE的评估指标（如准确率、F1分数）无法完全捕捉模型在细粒度语言现象（如否定、量化）上的表现，亟需更精细的诊断性测试来揭示模型短板。

常用场景

经典使用场景

GLUE基准测试作为自然语言理解领域最具影响力的综合性评估平台，其核心应用在于多任务文本分类的标准化评测。该数据集整合了涵盖语言可接受性判断（CoLA）、情感分析（SST-2）、语义等价性识别（MRPC、QQP）、自然语言推理（MNLI、RTE、WNLI）、问答匹配（QNLI）以及语义文本相似度计算（STS-B）等多项经典任务。研究者通常利用GLUE对预训练语言模型进行全面的泛化能力评估，通过在同一框架下比较模型在不同子任务上的表现，揭示模型在句法、语义和语用层面的理解深度。这种多维度、多任务的评测范式已成为衡量模型语言理解能力的黄金标准，为后续研究提供了可复现的参照基准。

解决学术问题

GLUE数据集深刻回应了自然语言处理领域长期存在的碎片化评估困境。在GLUE问世之前，不同研究团队往往采用各自独立的测试集和评价指标，导致模型性能难以横向比较，也阻碍了通用语言理解能力的系统性进步。GLUE通过构建统一的评测框架，解决了跨任务性能对比缺乏标准化基准的学术难题，使得研究者能够精准识别模型在特定语言现象上的优势与短板。该数据集的重要意义在于推动了预训练语言模型的范式革新，尤其是BERT等模型在GLUE排行榜上的优异表现，证实了大规模无监督预训练结合多任务微调的有效性，从而深刻改变了自然语言处理的研究方向与技术路线。

衍生相关工作

GLUE数据集的诞生催生了一系列具有里程碑意义的学术工作。BERT模型正是在GLUE基准上首次展示了双向Transformer预训练在多任务评估中的统治性表现，其提出的掩码语言模型与下一句预测任务成为后续研究的范本。紧随其后的RoBERTa通过优化训练策略与动态掩码机制，在GLUE上取得了更优成绩，揭示了预训练数据量与训练时长对模型性能的关键影响。XLNet与ALBERT等模型分别从自回归预训练和参数共享角度对GLUE任务进行了深入探索。SuperGLUE作为GLUE的进阶版本，针对原始基准中任务难度不足的问题，引入了更富挑战性的子任务，进一步推动了自然语言理解能力的边界拓展。