Fairlex

我们提出了一个由四个数据集组成的基准套件，用于评估预先训练的法律语言模型的公平性，以及用于为下游任务微调它们的技术。我们的基准涵盖四个司法管辖区（欧洲理事会、美国、瑞士和中国）、五种语言（英语、德语、法语、意大利语和中文）以及五个属性（性别、年龄、国籍/地区、语言和法律领域）的公平性。在我们的实验中，我们使用几种组鲁棒微调技术评估了预训练的语言模型，并表明在许多情况下，性能组差异是活跃的，而这些技术都不能保证公平性，也不能始终如一地减轻组差异。此外，我们还对结果进行了定量和定性分析，强调了在法律NLP中开发鲁棒性方法的开放挑战。

We propose a benchmark suite composed of four datasets for evaluating the fairness of pre-trained legal language models and the techniques for fine-tuning them for downstream tasks. Our benchmark covers fairness across four jurisdictions (the Council of Europe, the United States, Switzerland, and China), five languages (English, German, French, Italian, and Chinese), and five attributes (gender, age, nationality/region, language, and legal domain). In our experiments, we evaluated pre-trained legal language models using several group-robust fine-tuning techniques, and demonstrated that in many cases performance group disparities are prevalent, while none of these techniques can guarantee fairness or consistently mitigate such disparities. Furthermore, we conducted both quantitative and qualitative analyses of the results, highlighting the open challenges in developing robust methods for legal NLP.

搜集汇总

背景与挑战

背景概述

Fairlex是一个多语言基准数据集，用于评估法律文本处理中预训练语言模型的公平性。它覆盖四个司法管辖区、五种语言和五个属性，通过实验分析组性能差异，并强调现有技术难以保证公平性的挑战。