coastalcph/lex_glue
收藏Hugging Face2024-01-04 更新2024-03-04 收录
下载链接：
https://hf-mirror.com/datasets/coastalcph/lex_glue
下载链接
链接失效反馈官方服务：
资源简介：
---
annotations_creators:
- found
language_creators:
- found
language:
- en
license:
- cc-by-4.0
multilinguality:
- monolingual
size_categories:
- 10K<n<100K
source_datasets:
- extended
task_categories:
- question-answering
- text-classification
task_ids:
- multi-class-classification
- multi-label-classification
- multiple-choice-qa
- topic-classification
pretty_name: LexGLUE
config_names:
- case_hold
- ecthr_a
- ecthr_b
- eurlex
- ledgar
- scotus
- unfair_tos
dataset_info:
- config_name: case_hold
features:
- name: context
dtype: string
- name: endings
sequence: string
- name: label
dtype:
class_label:
names:
'0': '0'
'1': '1'
'2': '2'
'3': '3'
'4': '4'
splits:
- name: train
num_bytes: 74781706
num_examples: 45000
- name: test
num_bytes: 5989952
num_examples: 3600
- name: validation
num_bytes: 6474603
num_examples: 3900
download_size: 47303537
dataset_size: 87246261
- config_name: ecthr_a
features:
- name: text
sequence: string
- name: labels
sequence:
class_label:
names:
'0': '2'
'1': '3'
'2': '5'
'3': '6'
'4': '8'
'5': '9'
'6': '10'
'7': '11'
'8': '14'
'9': P1-1
splits:
- name: train
num_bytes: 89637449
num_examples: 9000
- name: test
num_bytes: 11884168
num_examples: 1000
- name: validation
num_bytes: 10985168
num_examples: 1000
download_size: 53352586
dataset_size: 112506785
- config_name: ecthr_b
features:
- name: text
sequence: string
- name: labels
sequence:
class_label:
names:
'0': '2'
'1': '3'
'2': '5'
'3': '6'
'4': '8'
'5': '9'
'6': '10'
'7': '11'
'8': '14'
'9': P1-1
splits:
- name: train
num_bytes: 89657649
num_examples: 9000
- name: test
num_bytes: 11886928
num_examples: 1000
- name: validation
num_bytes: 10987816
num_examples: 1000
download_size: 53352494
dataset_size: 112532393
- config_name: eurlex
features:
- name: text
dtype: string
- name: labels
sequence:
class_label:
names:
'0': '100163'
'1': '100168'
'2': '100169'
'3': '100170'
'4': '100171'
'5': '100172'
'6': '100173'
'7': '100174'
'8': '100175'
'9': '100176'
'10': '100177'
'11': '100179'
'12': '100180'
'13': '100183'
'14': '100184'
'15': '100185'
'16': '100186'
'17': '100187'
'18': '100189'
'19': '100190'
'20': '100191'
'21': '100192'
'22': '100193'
'23': '100194'
'24': '100195'
'25': '100196'
'26': '100197'
'27': '100198'
'28': '100199'
'29': '100200'
'30': '100201'
'31': '100202'
'32': '100204'
'33': '100205'
'34': '100206'
'35': '100207'
'36': '100212'
'37': '100214'
'38': '100215'
'39': '100220'
'40': '100221'
'41': '100222'
'42': '100223'
'43': '100224'
'44': '100226'
'45': '100227'
'46': '100229'
'47': '100230'
'48': '100231'
'49': '100232'
'50': '100233'
'51': '100234'
'52': '100235'
'53': '100237'
'54': '100238'
'55': '100239'
'56': '100240'
'57': '100241'
'58': '100242'
'59': '100243'
'60': '100244'
'61': '100245'
'62': '100246'
'63': '100247'
'64': '100248'
'65': '100249'
'66': '100250'
'67': '100252'
'68': '100253'
'69': '100254'
'70': '100255'
'71': '100256'
'72': '100257'
'73': '100258'
'74': '100259'
'75': '100260'
'76': '100261'
'77': '100262'
'78': '100263'
'79': '100264'
'80': '100265'
'81': '100266'
'82': '100268'
'83': '100269'
'84': '100270'
'85': '100271'
'86': '100272'
'87': '100273'
'88': '100274'
'89': '100275'
'90': '100276'
'91': '100277'
'92': '100278'
'93': '100279'
'94': '100280'
'95': '100281'
'96': '100282'
'97': '100283'
'98': '100284'
'99': '100285'
splits:
- name: train
num_bytes: 390770241
num_examples: 55000
- name: test
num_bytes: 59739094
num_examples: 5000
- name: validation
num_bytes: 41544476
num_examples: 5000
download_size: 208028049
dataset_size: 492053811
- config_name: ledgar
features:
- name: text
dtype: string
- name: label
dtype:
class_label:
names:
'0': Adjustments
'1': Agreements
'2': Amendments
'3': Anti-Corruption Laws
'4': Applicable Laws
'5': Approvals
'6': Arbitration
'7': Assignments
'8': Assigns
'9': Authority
'10': Authorizations
'11': Base Salary
'12': Benefits
'13': Binding Effects
'14': Books
'15': Brokers
'16': Capitalization
'17': Change In Control
'18': Closings
'19': Compliance With Laws
'20': Confidentiality
'21': Consent To Jurisdiction
'22': Consents
'23': Construction
'24': Cooperation
'25': Costs
'26': Counterparts
'27': Death
'28': Defined Terms
'29': Definitions
'30': Disability
'31': Disclosures
'32': Duties
'33': Effective Dates
'34': Effectiveness
'35': Employment
'36': Enforceability
'37': Enforcements
'38': Entire Agreements
'39': Erisa
'40': Existence
'41': Expenses
'42': Fees
'43': Financial Statements
'44': Forfeitures
'45': Further Assurances
'46': General
'47': Governing Laws
'48': Headings
'49': Indemnifications
'50': Indemnity
'51': Insurances
'52': Integration
'53': Intellectual Property
'54': Interests
'55': Interpretations
'56': Jurisdictions
'57': Liens
'58': Litigations
'59': Miscellaneous
'60': Modifications
'61': No Conflicts
'62': No Defaults
'63': No Waivers
'64': Non-Disparagement
'65': Notices
'66': Organizations
'67': Participations
'68': Payments
'69': Positions
'70': Powers
'71': Publicity
'72': Qualifications
'73': Records
'74': Releases
'75': Remedies
'76': Representations
'77': Sales
'78': Sanctions
'79': Severability
'80': Solvency
'81': Specific Performance
'82': Submission To Jurisdiction
'83': Subsidiaries
'84': Successors
'85': Survival
'86': Tax Withholdings
'87': Taxes
'88': Terminations
'89': Terms
'90': Titles
'91': Transactions With Affiliates
'92': Use Of Proceeds
'93': Vacations
'94': Venues
'95': Vesting
'96': Waiver Of Jury Trials
'97': Waivers
'98': Warranties
'99': Withholdings
splits:
- name: train
num_bytes: 43358291
num_examples: 60000
- name: test
num_bytes: 6845581
num_examples: 10000
- name: validation
num_bytes: 7143588
num_examples: 10000
download_size: 27650585
dataset_size: 57347460
- config_name: scotus
features:
- name: text
dtype: string
- name: label
dtype:
class_label:
names:
'0': '1'
'1': '2'
'2': '3'
'3': '4'
'4': '5'
'5': '6'
'6': '7'
'7': '8'
'8': '9'
'9': '10'
'10': '11'
'11': '12'
'12': '13'
splits:
- name: train
num_bytes: 178959316
num_examples: 5000
- name: test
num_bytes: 76213279
num_examples: 1400
- name: validation
num_bytes: 75600243
num_examples: 1400
download_size: 173411399
dataset_size: 330772838
- config_name: unfair_tos
features:
- name: text
dtype: string
- name: labels
sequence:
class_label:
names:
'0': Limitation of liability
'1': Unilateral termination
'2': Unilateral change
'3': Content removal
'4': Contract by using
'5': Choice of law
'6': Jurisdiction
'7': Arbitration
splits:
- name: train
num_bytes: 1041782
num_examples: 5532
- name: test
num_bytes: 303099
num_examples: 1607
- name: validation
num_bytes: 452111
num_examples: 2275
download_size: 865604
dataset_size: 1796992
configs:
- config_name: case_hold
data_files:
- split: train
path: case_hold/train-*
- split: test
path: case_hold/test-*
- split: validation
path: case_hold/validation-*
- config_name: ecthr_a
data_files:
- split: train
path: ecthr_a/train-*
- split: test
path: ecthr_a/test-*
- split: validation
path: ecthr_a/validation-*
- config_name: ecthr_b
data_files:
- split: train
path: ecthr_b/train-*
- split: test
path: ecthr_b/test-*
- split: validation
path: ecthr_b/validation-*
- config_name: eurlex
data_files:
- split: train
path: eurlex/train-*
- split: test
path: eurlex/test-*
- split: validation
path: eurlex/validation-*
- config_name: ledgar
data_files:
- split: train
path: ledgar/train-*
- split: test
path: ledgar/test-*
- split: validation
path: ledgar/validation-*
- config_name: scotus
data_files:
- split: train
path: scotus/train-*
- split: test
path: scotus/test-*
- split: validation
path: scotus/validation-*
- config_name: unfair_tos
data_files:
- split: train
path: unfair_tos/train-*
- split: test
path: unfair_tos/test-*
- split: validation
path: unfair_tos/validation-*
---
# Dataset Card for "LexGLUE"
## Table of Contents
- [Dataset Description](#dataset-description)
- [Dataset Summary](#dataset-summary)
- [Supported Tasks and Leaderboards](#supported-tasks-and-leaderboards)
- [Languages](#languages)
- [Dataset Structure](#dataset-structure)
- [Data Instances](#data-instances)
- [Data Fields](#data-fields)
- [Data Splits](#data-splits)
- [Dataset Creation](#dataset-creation)
- [Curation Rationale](#curation-rationale)
- [Source Data](#source-data)
- [Annotations](#annotations)
- [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
- [Social Impact of Dataset](#social-impact-of-dataset)
- [Discussion of Biases](#discussion-of-biases)
- [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
- [Dataset Curators](#dataset-curators)
- [Licensing Information](#licensing-information)
- [Citation Information](#citation-information)
- [Contributions](#contributions)
## Dataset Description
- **Homepage:** https://github.com/coastalcph/lex-glue
- **Repository:** https://github.com/coastalcph/lex-glue
- **Paper:** https://arxiv.org/abs/2110.00976
- **Leaderboard:** https://github.com/coastalcph/lex-glue
- **Point of Contact:** [Ilias Chalkidis](mailto:ilias.chalkidis@di.ku.dk)
### Dataset Summary
Inspired by the recent widespread use of the GLUE multi-task benchmark NLP dataset (Wang et al., 2018), the subsequent more difficult SuperGLUE (Wang et al., 2019), other previous multi-task NLP benchmarks (Conneau and Kiela, 2018; McCann et al., 2018), and similar initiatives in other domains (Peng et al., 2019), we introduce the *Legal General Language Understanding Evaluation (LexGLUE) benchmark*, a benchmark dataset to evaluate the performance of NLP methods in legal tasks. LexGLUE is based on seven existing legal NLP datasets, selected using criteria largely from SuperGLUE.
As in GLUE and SuperGLUE (Wang et al., 2019b,a), one of our goals is to push towards generic (or ‘foundation’) models that can cope with multiple NLP tasks, in our case legal NLP tasks possibly with limited task-specific fine-tuning. Another goal is to provide a convenient and informative entry point for NLP researchers and practitioners wishing to explore or develop methods for legalNLP. Having these goals in mind, the datasets we include in LexGLUE and the tasks they address have been simplified in several ways to make it easier for newcomers and generic models to address all tasks.
LexGLUE benchmark is accompanied by experimental infrastructure that relies on Hugging Face Transformers library and resides at: https://github.com/coastalcph/lex-glue.
### Supported Tasks and Leaderboards
The supported tasks are the following:
<table>
<tr><td>Dataset</td><td>Source</td><td>Sub-domain</td><td>Task Type</td><td>Classes</td><tr>
<tr><td>ECtHR (Task A)</td><td> <a href="https://aclanthology.org/P19-1424/">Chalkidis et al. (2019)</a> </td><td>ECHR</td><td>Multi-label classification</td><td>10+1</td></tr>
<tr><td>ECtHR (Task B)</td><td> <a href="https://aclanthology.org/2021.naacl-main.22/">Chalkidis et al. (2021a)</a> </td><td>ECHR</td><td>Multi-label classification </td><td>10+1</td></tr>
<tr><td>SCOTUS</td><td> <a href="http://scdb.wustl.edu">Spaeth et al. (2020)</a></td><td>US Law</td><td>Multi-class classification</td><td>14</td></tr>
<tr><td>EUR-LEX</td><td> <a href="https://arxiv.org/abs/2109.00904">Chalkidis et al. (2021b)</a></td><td>EU Law</td><td>Multi-label classification</td><td>100</td></tr>
<tr><td>LEDGAR</td><td> <a href="https://aclanthology.org/2020.lrec-1.155/">Tuggener et al. (2020)</a></td><td>Contracts</td><td>Multi-class classification</td><td>100</td></tr>
<tr><td>UNFAIR-ToS</td><td><a href="https://arxiv.org/abs/1805.01217"> Lippi et al. (2019)</a></td><td>Contracts</td><td>Multi-label classification</td><td>8+1</td></tr>
<tr><td>CaseHOLD</td><td><a href="https://arxiv.org/abs/2104.08671">Zheng et al. (2021)</a></td><td>US Law</td><td>Multiple choice QA</td><td>n/a</td></tr>
</table>
#### ecthr_a
The European Court of Human Rights (ECtHR) hears allegations that a state has breached human rights provisions of the European Convention of Human Rights (ECHR). For each case, the dataset provides a list of factual paragraphs (facts) from the case description. Each case is mapped to articles of the ECHR that were violated (if any).
#### ecthr_b
The European Court of Human Rights (ECtHR) hears allegations that a state has breached human rights provisions of the European Convention of Human Rights (ECHR). For each case, the dataset provides a list of factual paragraphs (facts) from the case description. Each case is mapped to articles of ECHR that were allegedly violated (considered by the court).
#### scotus
The US Supreme Court (SCOTUS) is the highest federal court in the United States of America and generally hears only the most controversial or otherwise complex cases which have not been sufficiently well solved by lower courts. This is a single-label multi-class classification task, where given a document (court opinion), the task is to predict the relevant issue areas. The 14 issue areas cluster 278 issues whose focus is on the subject matter of the controversy (dispute).
#### eurlex
European Union (EU) legislation is published in EUR-Lex portal. All EU laws are annotated by EU's Publications Office with multiple concepts from the EuroVoc thesaurus, a multilingual thesaurus maintained by the Publications Office. The current version of EuroVoc contains more than 7k concepts referring to various activities of the EU and its Member States (e.g., economics, health-care, trade). Given a document, the task is to predict its EuroVoc labels (concepts).
#### ledgar
LEDGAR dataset aims contract provision (paragraph) classification. The contract provisions come from contracts obtained from the US Securities and Exchange Commission (SEC) filings, which are publicly available from EDGAR. Each label represents the single main topic (theme) of the corresponding contract provision.
#### unfair_tos
The UNFAIR-ToS dataset contains 50 Terms of Service (ToS) from on-line platforms (e.g., YouTube, Ebay, Facebook, etc.). The dataset has been annotated on the sentence-level with 8 types of unfair contractual terms (sentences), meaning terms that potentially violate user rights according to the European consumer law.
#### case_hold
The CaseHOLD (Case Holdings on Legal Decisions) dataset includes multiple choice questions about holdings of US court cases from the Harvard Law Library case law corpus. Holdings are short summaries of legal rulings accompany referenced decisions relevant for the present case. The input consists of an excerpt (or prompt) from a court decision, containing a reference to a particular case, while the holding statement is masked out. The model must identify the correct (masked) holding statement from a selection of five choices.
The current leaderboard includes several Transformer-based (Vaswaniet al., 2017) pre-trained language models, which achieve state-of-the-art performance in most NLP tasks (Bommasani et al., 2021) and NLU benchmarks (Wang et al., 2019a). Results reported by [Chalkidis et al. (2021)](https://arxiv.org/abs/2110.00976):
*Task-wise Test Results*
<table>
<tr><td><b>Dataset</b></td><td><b>ECtHR A</b></td><td><b>ECtHR B</b></td><td><b>SCOTUS</b></td><td><b>EUR-LEX</b></td><td><b>LEDGAR</b></td><td><b>UNFAIR-ToS</b></td><td><b>CaseHOLD</b></td></tr>
<tr><td><b>Model</b></td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1</td><td>μ-F1 / m-F1 </td></tr>
<tr><td>TFIDF+SVM</td><td> 64.7 / 51.7 </td><td>74.6 / 65.1 </td><td> <b>78.2</b> / <b>69.5</b> </td><td>71.3 / 51.4 </td><td>87.2 / 82.4 </td><td>95.4 / 78.8</td><td>n/a </td></tr>
<tr><td colspan="8" style='text-align:center'><b>Medium-sized Models (L=12, H=768, A=12)</b></td></tr>
<td>BERT</td> <td> 71.2 / 63.6 </td> <td> 79.7 / 73.4 </td> <td> 68.3 / 58.3 </td> <td> 71.4 / 57.2 </td> <td> 87.6 / 81.8 </td> <td> 95.6 / 81.3 </td> <td> 70.8 </td> </tr>
<td>RoBERTa</td> <td> 69.2 / 59.0 </td> <td> 77.3 / 68.9 </td> <td> 71.6 / 62.0 </td> <td> 71.9 / <b>57.9</b> </td> <td> 87.9 / 82.3 </td> <td> 95.2 / 79.2 </td> <td> 71.4 </td> </tr>
<td>DeBERTa</td> <td> 70.0 / 60.8 </td> <td> 78.8 / 71.0 </td> <td> 71.1 / 62.7 </td> <td> <b>72.1</b> / 57.4 </td> <td> 88.2 / 83.1 </td> <td> 95.5 / 80.3 </td> <td> 72.6 </td> </tr>
<td>Longformer</td> <td> 69.9 / 64.7 </td> <td> 79.4 / 71.7 </td> <td> 72.9 / 64.0 </td> <td> 71.6 / 57.7 </td> <td> 88.2 / 83.0 </td> <td> 95.5 / 80.9 </td> <td> 71.9 </td> </tr>
<td>BigBird</td> <td> 70.0 / 62.9 </td> <td> 78.8 / 70.9 </td> <td> 72.8 / 62.0 </td> <td> 71.5 / 56.8 </td> <td> 87.8 / 82.6 </td> <td> 95.7 / 81.3 </td> <td> 70.8 </td> </tr>
<td>Legal-BERT</td> <td> 70.0 / 64.0 </td> <td> <b>80.4</b> / <b>74.7</b> </td> <td> 76.4 / 66.5 </td> <td> <b>72.1</b> / 57.4 </td> <td> 88.2 / 83.0 </td> <td> <b>96.0</b> / <b>83.0</b> </td> <td> 75.3 </td> </tr>
<td>CaseLaw-BERT</td> <td> 69.8 / 62.9 </td> <td> 78.8 / 70.3 </td> <td> 76.6 / 65.9 </td> <td> 70.7 / 56.6 </td> <td> 88.3 / 83.0 </td> <td> <b>96.0</b> / 82.3 </td> <td> <b>75.4</b> </td> </tr>
<tr><td colspan="8" style='text-align:center'><b>Large-sized Models (L=24, H=1024, A=18)</b></td></tr>
<tr><td>RoBERTa</td> <td> <b>73.8</b> / <b>67.6</b> </td> <td> 79.8 / 71.6 </td> <td> 75.5 / 66.3 </td> <td> 67.9 / 50.3 </td> <td> <b>88.6</b> / <b>83.6</b> </td> <td> 95.8 / 81.6 </td> <td> 74.4 </td> </tr>
</table>
*Averaged (Mean over Tasks) Test Results*
<table>
<tr><td><b>Averaging</b></td><td><b>Arithmetic</b></td><td><b>Harmonic</b></td><td><b>Geometric</b></td></tr>
<tr><td><b>Model</b></td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1 </td><td>μ-F1 / m-F1 </td></tr>
<tr><td colspan="4" style='text-align:center'><b>Medium-sized Models (L=12, H=768, A=12)</b></td></tr>
<tr><td>BERT</td><td> 77.8 / 69.5 </td><td> 76.7 / 68.2 </td><td> 77.2 / 68.8 </td></tr>
<tr><td>RoBERTa</td><td> 77.8 / 68.7 </td><td> 76.8 / 67.5 </td><td> 77.3 / 68.1 </td></tr>
<tr><td>DeBERTa</td><td> 78.3 / 69.7 </td><td> 77.4 / 68.5 </td><td> 77.8 / 69.1 </td></tr>
<tr><td>Longformer</td><td> 78.5 / 70.5 </td><td> 77.5 / 69.5 </td><td> 78.0 / 70.0 </td></tr>
<tr><td>BigBird</td><td> 78.2 / 69.6 </td><td> 77.2 / 68.5 </td><td> 77.7 / 69.0 </td></tr>
<tr><td>Legal-BERT</td><td> <b>79.8</b> / <b>72.0</b> </td><td> <b>78.9</b> / <b>70.8</b> </td><td> <b>79.3</b> / <b>71.4</b> </td></tr>
<tr><td>CaseLaw-BERT</td><td> 79.4 / 70.9 </td><td> 78.5 / 69.7 </td><td> 78.9 / 70.3 </td></tr>
<tr><td colspan="4" style='text-align:center'><b>Large-sized Models (L=24, H=1024, A=18)</b></td></tr>
<tr><td>RoBERTa</td><td> 79.4 / 70.8 </td><td> 78.4 / 69.1 </td><td> 78.9 / 70.0 </td></tr>
</table>
### Languages
We only consider English datasets, to make experimentation easier for researchers across the globe.
## Dataset Structure
### Data Instances
#### ecthr_a
An example of 'train' looks as follows.
```json
{
"text": ["8. The applicant was arrested in the early morning of 21 October 1990 ...", ...],
"labels": [6]
}
```
#### ecthr_b
An example of 'train' looks as follows.
```json
{
"text": ["8. The applicant was arrested in the early morning of 21 October 1990 ...", ...],
"label": [5, 6]
}
```
#### scotus
An example of 'train' looks as follows.
```json
{
"text": "Per Curiam\nSUPREME COURT OF THE UNITED STATES\nRANDY WHITE, WARDEN v. ROGER L. WHEELER\n Decided December 14, 2015\nPER CURIAM.\nA death sentence imposed by a Kentucky trial court and\naffirmed by the ...",
"label": 8
}
```
#### eurlex
An example of 'train' looks as follows.
```json
{
"text": "COMMISSION REGULATION (EC) No 1629/96 of 13 August 1996 on an invitation to tender for the refund on export of wholly milled round grain rice to certain third countries ...",
"labels": [4, 20, 21, 35, 68]
}
```
#### ledgar
An example of 'train' looks as follows.
```json
{
"text": "All Taxes shall be the financial responsibility of the party obligated to pay such Taxes as determined by applicable law and neither party is or shall be liable at any time for any of the other party ...",
"label": 32
}
```
#### unfair_tos
An example of 'train' looks as follows.
```json
{
"text": "tinder may terminate your account at any time without notice if it believes that you have violated this agreement.",
"label": 2
}
```
#### casehold
An example of 'test' looks as follows.
```json
{
"context": "In Granato v. City and County of Denver, No. CIV 11-0304 MSK/BNB, 2011 WL 3820730 (D.Colo. Aug. 20, 2011), the Honorable Marcia S. Krieger, now-Chief United States District Judge for the District of Colorado, ruled similarly: At a minimum, a party asserting a Mo-nell claim must plead sufficient facts to identify ... to act pursuant to City or State policy, custom, decision, ordinance, re d 503, 506-07 (3d Cir.l985)(<HOLDING>).",
"endings": ["holding that courts are to accept allegations in the complaint as being true including monell policies and writing that a federal court reviewing the sufficiency of a complaint has a limited task",
"holding that for purposes of a class certification motion the court must accept as true all factual allegations in the complaint and may draw reasonable inferences therefrom",
"recognizing that the allegations of the complaint must be accepted as true on a threshold motion to dismiss",
"holding that a court need not accept as true conclusory allegations which are contradicted by documents referred to in the complaint",
"holding that where the defendant was in default the district court correctly accepted the fact allegations of the complaint as true"
],
"label": 0
}
```
### Data Fields
#### ecthr_a
- `text`: a list of `string` features (list of factual paragraphs (facts) from the case description).
- `labels`: a list of classification labels (a list of violated ECHR articles, if any) .
<details>
<summary>List of ECHR articles</summary>
"Article 2", "Article 3", "Article 5", "Article 6", "Article 8", "Article 9", "Article 10", "Article 11", "Article 14", "Article 1 of Protocol 1"
</details>
#### ecthr_b
- `text`: a list of `string` features (list of factual paragraphs (facts) from the case description)
- `labels`: a list of classification labels (a list of articles considered).
<details>
<summary>List of ECHR articles</summary>
"Article 2", "Article 3", "Article 5", "Article 6", "Article 8", "Article 9", "Article 10", "Article 11", "Article 14", "Article 1 of Protocol 1"
</details>
#### scotus
- `text`: a `string` feature (the court opinion).
- `label`: a classification label (the relevant issue area).
<details>
<summary>List of issue areas</summary>
(1, Criminal Procedure), (2, Civil Rights), (3, First Amendment), (4, Due Process), (5, Privacy), (6, Attorneys), (7, Unions), (8, Economic Activity), (9, Judicial Power), (10, Federalism), (11, Interstate Relations), (12, Federal Taxation), (13, Miscellaneous), (14, Private Action)
</details>
#### eurlex
- `text`: a `string` feature (an EU law).
- `labels`: a list of classification labels (a list of relevant EUROVOC concepts).
<details>
<summary>List of EUROVOC concepts</summary>
The list is very long including 100 EUROVOC concepts. You can find the EUROVOC concepts descriptors <a href="https://raw.githubusercontent.com/nlpaueb/multi-eurlex/master/data/eurovoc_descriptors.json">here</a>.
</details>
#### ledgar
- `text`: a `string` feature (a contract provision/paragraph).
- `label`: a classification label (the type of contract provision).
<details>
<summary>List of contract provision types</summary>
"Adjustments", "Agreements", "Amendments", "Anti-Corruption Laws", "Applicable Laws", "Approvals", "Arbitration", "Assignments", "Assigns", "Authority", "Authorizations", "Base Salary", "Benefits", "Binding Effects", "Books", "Brokers", "Capitalization", "Change In Control", "Closings", "Compliance With Laws", "Confidentiality", "Consent To Jurisdiction", "Consents", "Construction", "Cooperation", "Costs", "Counterparts", "Death", "Defined Terms", "Definitions", "Disability", "Disclosures", "Duties", "Effective Dates", "Effectiveness", "Employment", "Enforceability", "Enforcements", "Entire Agreements", "Erisa", "Existence", "Expenses", "Fees", "Financial Statements", "Forfeitures", "Further Assurances", "General", "Governing Laws", "Headings", "Indemnifications", "Indemnity", "Insurances", "Integration", "Intellectual Property", "Interests", "Interpretations", "Jurisdictions", "Liens", "Litigations", "Miscellaneous", "Modifications", "No Conflicts", "No Defaults", "No Waivers", "Non-Disparagement", "Notices", "Organizations", "Participations", "Payments", "Positions", "Powers", "Publicity", "Qualifications", "Records", "Releases", "Remedies", "Representations", "Sales", "Sanctions", "Severability", "Solvency", "Specific Performance", "Submission To Jurisdiction", "Subsidiaries", "Successors", "Survival", "Tax Withholdings", "Taxes", "Terminations", "Terms", "Titles", "Transactions With Affiliates", "Use Of Proceeds", "Vacations", "Venues", "Vesting", "Waiver Of Jury Trials", "Waivers", "Warranties", "Withholdings",
</details>
#### unfair_tos
- `text`: a `string` feature (a ToS sentence)
- `labels`: a list of classification labels (a list of unfair types, if any).
<details>
<summary>List of unfair types</summary>
"Limitation of liability", "Unilateral termination", "Unilateral change", "Content removal", "Contract by using", "Choice of law", "Jurisdiction", "Arbitration"
</details>
#### casehold
- `context`: a `string` feature (a context sentence incl. a masked holding statement).
- `holdings`: a list of `string` features (a list of candidate holding statements).
- `label`: a classification label (the id of the original/correct holding).
### Data Splits
<table>
<tr><td>Dataset </td><td>Training</td><td>Development</td><td>Test</td><td>Total</td></tr>
<tr><td>ECtHR (Task A)</td><td>9,000</td><td>1,000</td><td>1,000</td><td>11,000</td></tr>
<tr><td>ECtHR (Task B)</td><td>9,000</td><td>1,000</td><td>1,000</td><td>11,000</td></tr>
<tr><td>SCOTUS</td><td>5,000</td><td>1,400</td><td>1,400</td><td>7,800</td></tr>
<tr><td>EUR-LEX</td><td>55,000</td><td>5,000</td><td>5,000</td><td>65,000</td></tr>
<tr><td>LEDGAR</td><td>60,000</td><td>10,000</td><td>10,000</td><td>80,000</td></tr>
<tr><td>UNFAIR-ToS</td><td>5,532</td><td>2,275</td><td>1,607</td><td>9,414</td></tr>
<tr><td>CaseHOLD</td><td>45,000</td><td>3,900</td><td>3,900</td><td>52,800</td></tr>
</table>
## Dataset Creation
### Curation Rationale
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Source Data
<table>
<tr><td>Dataset</td><td>Source</td><td>Sub-domain</td><td>Task Type</td><tr>
<tr><td>ECtHR (Task A)</td><td> <a href="https://aclanthology.org/P19-1424/">Chalkidis et al. (2019)</a> </td><td>ECHR</td><td>Multi-label classification</td></tr>
<tr><td>ECtHR (Task B)</td><td> <a href="https://aclanthology.org/2021.naacl-main.22/">Chalkidis et al. (2021a)</a> </td><td>ECHR</td><td>Multi-label classification </td></tr>
<tr><td>SCOTUS</td><td> <a href="http://scdb.wustl.edu">Spaeth et al. (2020)</a></td><td>US Law</td><td>Multi-class classification</td></tr>
<tr><td>EUR-LEX</td><td> <a href="https://arxiv.org/abs/2109.00904">Chalkidis et al. (2021b)</a></td><td>EU Law</td><td>Multi-label classification</td></tr>
<tr><td>LEDGAR</td><td> <a href="https://aclanthology.org/2020.lrec-1.155/">Tuggener et al. (2020)</a></td><td>Contracts</td><td>Multi-class classification</td></tr>
<tr><td>UNFAIR-ToS</td><td><a href="https://arxiv.org/abs/1805.01217"> Lippi et al. (2019)</a></td><td>Contracts</td><td>Multi-label classification</td></tr>
<tr><td>CaseHOLD</td><td><a href="https://arxiv.org/abs/2104.08671">Zheng et al. (2021)</a></td><td>US Law</td><td>Multiple choice QA</td></tr>
</table>
#### Initial Data Collection and Normalization
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
#### Who are the source language producers?
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Annotations
#### Annotation process
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
#### Who are the annotators?
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Personal and Sensitive Information
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## Considerations for Using the Data
### Social Impact of Dataset
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Discussion of Biases
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Other Known Limitations
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
## Additional Information
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Dataset Curators
*Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras.*
*LexGLUE: A Benchmark Dataset for Legal Language Understanding in English.*
*2022. In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland.*
### Licensing Information
[More Information Needed](https://github.com/huggingface/datasets/blob/master/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)
### Citation Information
[*Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras.*
*LexGLUE: A Benchmark Dataset for Legal Language Understanding in English.*
*2022. In the Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland.*](https://arxiv.org/abs/2110.00976)
```
@inproceedings{chalkidis-etal-2021-lexglue,
title={LexGLUE: A Benchmark Dataset for Legal Language Understanding in English},
author={Chalkidis, Ilias and Jana, Abhik and Hartung, Dirk and
Bommarito, Michael and Androutsopoulos, Ion and Katz, Daniel Martin and
Aletras, Nikolaos},
year={2022},
booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics},
address={Dubln, Ireland},
}
```
### Contributions
Thanks to [@iliaschalkidis](https://github.com/iliaschalkidis) for adding this dataset.
提供机构：
coastalcph
原始信息汇总
数据集概述
数据集名称： LexGLUE
语言： 英语（en）
许可证： CC-BY-4.0
多语言性： 单语种（monolingual）
大小类别： 10K<n<100K
源数据集： 扩展（extended）
任务类别：
- 问答（question-answering）
- 文本分类（text-classification）
任务ID：
- 多类分类（multi-class-classification）
- 多标签分类（multi-label-classification）
- 多选问答（multiple-choice-qa）
- 主题分类（topic-classification）
配置名称：
- case_hold
- ecthr_a
- ecthr_b
- eurlex
- ledgar
- scotus
- unfair_tos
数据集详细信息
case_hold
- 特征：
- context: string
- endings: sequence of string
- label: class_label with names 0 to 4
- 分割：
- train: 45000 examples, 74781706 bytes
- test: 3600 examples, 5989952 bytes
- validation: 3900 examples, 6474603 bytes
- 下载大小： 47303537 bytes
- 数据集大小： 87246261 bytes
ecthr_a
- 特征：
- text: sequence of string
- labels: class_label with names 2 to 14 and P1-1
- 分割：
- train: 9000 examples, 89637449 bytes
- test: 1000 examples, 11884168 bytes
- validation: 1000 examples, 10985168 bytes
- 下载大小： 53352586 bytes
- 数据集大小： 112506785 bytes
ecthr_b
- 特征：
- text: sequence of string
- labels: class_label with names 2 to 14 and P1-1
- 分割：
- train: 9000 examples, 89657649 bytes
- test: 1000 examples, 11886928 bytes
- validation: 1000 examples, 10987816 bytes
- 下载大小： 53352494 bytes
- 数据集大小： 112532393 bytes
eurlex
- 特征：
- text: string
- labels: class_label with names 100163 to 100285
- 分割：
- train: 55000 examples, 390770241 bytes
- test: 5000 examples, 59739094 bytes
- validation: 5000 examples, 41544476 bytes
- 下载大小： 208028049 bytes
- 数据集大小： 492053811 bytes
ledgar
- 特征：
- text: string
- label: class_label with names Adjustments to Withholdings
- 分割：
- train: 60000 examples, 43358291 bytes
- test: 10000 examples, 6845581 bytes
- validation: 10000 examples, 7143588 bytes
- 下载大小： 27650585 bytes
- 数据集大小： 57347460 bytes
scotus
- 特征：
- text: string
- label: class_label with names 1 to 13
- 分割：
- train: 5000 examples, 178959316 bytes
- test: 1400 examples, 76213279 bytes
- validation: 1400 examples, 75600243 bytes
- 下载大小： 173411399 bytes
- 数据集大小： 330772838 bytes
unfair_tos
- 特征：
- text: string
- labels: class_label with names Limitation of liability to Arbitration
- 分割：
- train: 5532 examples, 1041782 bytes
- test: 1607 examples, 303099 bytes
- validation: 2275 examples, 452111 bytes
- 下载大小： 865604 bytes
- 数据集大小： 1796992 bytes
搜集汇总
数据集介绍
构建方式
在自然语言处理领域，法律文本的复杂性对模型提出了独特挑战。LexGLUE基准的构建借鉴了GLUE与SuperGLUE的理念，通过精心筛选七个现有法律数据集整合而成。这些数据集源自欧洲人权法院案例、美国最高法院判决、欧盟法律文件、合同条款及服务协议等权威法律文献，涵盖了多标签分类、多类别分类及多项选择问答等多种任务类型。数据集的构建过程注重保留法律文本的原始结构与语义完整性，同时进行了适度简化以降低研究门槛，确保其既能反映法律语言的特质，又便于通用模型进行跨任务评估与学习。
使用方法
使用LexGLUE数据集时，研究者可通过Hugging Face平台直接加载特定子数据集配置，如`case_hold`或`eurlex`。每个子数据集以标准化的文本与标签字段组织，支持直接接入基于Transformer的预训练模型进行微调。实验流程通常包括利用训练集进行模型优化，在验证集上调整超参数，最终在测试集上评估性能指标如宏平均F1分数。数据集兼容常见的自然语言处理框架，并提供了与原始论文一致的评估脚本，便于结果复现与比较。用户可依据具体任务需求，灵活选择单一数据集进行深入探索，或在全部七个任务上开展跨领域综合评估，以推动通用法律智能模型的发展。
背景与挑战
背景概述
LexGLUE数据集由哥本哈根大学海岸计算研究团队于2021年提出，旨在构建一个法律领域的通用语言理解评估基准。该数据集整合了七个现有的法律自然语言处理数据集，涵盖欧洲人权法院案例、美国最高法院判决、欧盟法律文件、合同条款及服务协议等多个子领域。其核心研究问题在于推动通用模型在法律文本的多任务处理能力，为法律自然语言处理研究提供标准化的评估框架，显著促进了法律智能分析技术的发展与应用。
当前挑战
LexGLUE数据集面临的挑战主要体现在两个方面：其一，法律文本具有高度专业性和复杂性，模型需准确理解法律术语、逻辑结构及跨法域差异，以完成分类、问答等任务；其二，数据构建过程中需协调多源异构的法律文档，确保标注的一致性与法律合规性，同时处理长文本序列与多标签分类带来的计算与标注负担。
常用场景
经典使用场景
在法律自然语言处理领域，LexGLUE数据集作为综合性基准测试平台，其经典使用场景聚焦于评估与比较各类自然语言理解模型在多样化法律文本任务上的性能表现。该数据集整合了来自欧洲人权法院案例、美国最高法院判决、欧盟法律文件、合同条款以及服务协议等多个法律子领域的七项任务，涵盖了多标签分类、多类别分类及多项选择问答等多种任务类型。研究人员通过在此数据集上训练与测试模型，能够系统性地衡量模型在法律语义理解、条文关联分析以及判决预测等方面的泛化能力与鲁棒性，从而推动法律智能技术的标准化发展。
解决学术问题
LexGLUE数据集有效应对了法律自然语言处理研究中长期存在的任务分散与评估标准不统一的问题。通过构建一个多任务、跨法系的统一评估框架，该数据集使得研究者能够系统探索通用法律语言模型的潜力，缓解了以往因数据稀缺或领域特定性过强而导致的方法比较困难。其意义在于促进了法律文本理解技术的可复现性与可比性，为探索领域自适应、少样本学习以及长文本处理等前沿课题提供了扎实的实验基础，进而加速了人工智能与法律交叉学科的融合与创新。
实际应用
在实际应用层面，LexGLUE数据集支撑了多项法律科技产品的开发与优化。例如，基于该数据集训练的模型可辅助律师进行案例检索与案情预测，提升法律研究的效率；在合同审查场景中，模型能够自动识别条款类型与潜在的不公平条款，帮助企业合规部门进行风险管控；此外，在司法智能化建设中，该类技术可用于判决文书自动分类、法律条文推荐以及司法舆情分析，为法院和立法机构提供数据驱动的决策支持，从而提升法律服务的可及性与公正性。
数据集最近研究
最新研究方向
在人工智能与法律交叉领域，LexGLUE数据集作为法律自然语言处理的标准评估基准，正推动着前沿研究的深化。当前研究聚焦于开发能够处理长文本、多标签分类及复杂法律推理的预训练模型，例如基于Transformer架构的模型变体。这些模型通过微调策略适应不同法域的具体任务，如欧洲人权法院案例的违规预测、美国最高法院意见书的议题分类，以及欧盟法律文件的主题标注。随着法律科技应用的扩展，该数据集促进了模型在合同条款解析、不公平条款检测等实际场景中的性能提升，为法律智能化提供了关键的技术支撑。
以上内容由遇见数据集搜集并总结生成
