Machine learning & fairness: an integrated multicriteria approach for the evaluation of supervised classifiers

Does <i>Multiple Criteria Decision Aiding</i> (MCDA) improve the process of evaluating <i>Machine Learning</i> (ML) algorithms, when critical criteria of <i>fairness</i> are concurrently considered, beyond predictive power? To address this question, we employ several notions of <i>fairness,</i> such as <i>Demographic Parity</i>, <i>Equalized Odds</i>, and <i>Lack of Disparate Mistreatment</i>, and we appraise a set of supervised ML classifiers, under one of the most popular MCDA outranking methods, that is, the <i>Preference Ranking Organization Method for Enrichment Evaluation</i> (PROMETHEE) II. Moreover, to avoid any arbitrary choice in the importance attached to the criteria we apply the <i>Stochastic Multicriteria Acceptability Analysis</i> (SMAA), providing information in statistical terms through simulations. The empirical testing is processed over well-known databases, with several representative sub-datasets, securing variation in terms of observations’ volume. Overall, a series of ranking patterns that persists in the evaluation of the ML classifiers, across the utilized MCDA methodology and datasets, offers valuable relevant insights and documents specific useful interpretations. The obtained findings provide robust support that MCDA can be effectively exploited for the appraisal of ML classifiers, when aiming at the simultaneous consideration of critical fairness metrics, apart from the typical dimensions related to predictive power.

多准则决策辅助（Multiple Criteria Decision Aiding，MCDA）在同时考量公平性（fairness）关键准则且兼顾预测性能以外维度的前提下，能否优化机器学习（Machine Learning，ML）算法的评估流程？为解答这一问题，本研究采用了人口统计学均等（Demographic Parity）、均等赔率（Equalized Odds）与无差别对待（Lack of Disparate Mistreatment）等多种公平性定义，并依托当前主流的多准则决策辅助排序方法之一——偏好排序组织富集评估法（Preference Ranking Organization Method for Enrichment Evaluation，PROMETHEE）II，对一系列有监督机器学习分类器开展评估。此外，为规避准则权重赋值的主观任意性，本研究应用了随机多准则可接受性分析（Stochastic Multicriteria Acceptability Analysis，SMAA），通过模拟实验生成统计层面的分析结果。实证测试依托多个知名数据库及其代表性子数据集展开，确保样本体量存在差异。整体而言，在所采用的多准则决策辅助方法与数据集范围内，机器学习分类器评估中呈现出一系列稳定的排序模式，这为相关研究提供了极具价值的洞见，并形成了具体且实用的解读结论。本研究结果提供了坚实支撑，表明当需要同时考量公平性关键指标而非仅聚焦于预测性能相关的典型维度时，多准则决策辅助技术可被有效应用于机器学习分类器的评估工作。