LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

This benchmark dataset is published with the article: 

Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Martin Katz, and Nikolaos Aletras. 2021. LexGLUE: A Benchmark Dataset for Legal Language Understanding in English. ArXiv.

Short Description

Inspired by the recent widespread use of the GLUE multi-task benchmark NLP dataset (Wang et al., 2018), the subsequent more difficult SuperGLUE (Wang et al., 2019), other previous multi-task NLP benchmarks (Conneau and Kiela,2018; McCann et al., 2018), and similar initiatives in other domains (Peng et al.,  2019), we introduce LexGLUE, a benchmark dataset to evaluate the performance of NLP methods in legal tasks. LexGLUE is based on seven existing legal NLP datasets:

ECtHR Task A  (Chalkidis et al., 2019)

ECtHR Task B  (Chalkidis et al., 2021a)

SCOTUS (Spaeth et al., 2020)

EUR-LEX (Chalkidis et al., 2021b)

LEDGAR (Tuggener et al. (2020)

UNFAIR-ToS (Lippi et al., 2019)

CaseHOLD (Zheng et al., 2021)