Machine learning & fairness: an integrated multicriteria approach for the evaluation of supervised classifiers

Does Multiple Criteria Decision Aiding (MCDA) improve the process of evaluating Machine Learning (ML) algorithms, when critical criteria of fairness are concurrently considered, beyond predictive power? To address this question, we employ several notions of fairness, such as Demographic Parity, Equalized Odds, and Lack of Disparate Mistreatment, and we appraise a set of supervised ML classifiers, under one of the most popular MCDA outranking methods, that is, the Preference Ranking Organization Method for Enrichment Evaluation (PROMETHEE) II. Moreover, to avoid any arbitrary choice in the importance attached to the criteria we apply the Stochastic Multicriteria Acceptability Analysis (SMAA), providing information in statistical terms through simulations. The empirical testing is processed over well-known databases, with several representative sub-datasets, securing variation in terms of observations’ volume. Overall, a series of ranking patterns that persists in the evaluation of the ML classifiers, across the utilized MCDA methodology and datasets, offers valuable relevant insights and documents specific useful interpretations. The obtained findings provide robust support that MCDA can be effectively exploited for the appraisal of ML classifiers, when aiming at the simultaneous consideration of critical fairness metrics, apart from the typical dimensions related to predictive power.当同时考量公平性关键指标而非仅关注预测性能时，多准则决策辅助（Multiple Criteria Decision Aiding, MCDA）能否优化机器学习（Machine Learning, ML）算法的评估流程？为解答该问题，本研究采用了人口统计均等（Demographic Parity）、均等赔率（Equalized Odds）与无差异化误判（Lack of Disparate Mistreatment）等多种公平性定义，并借助当前主流的MCDA排序方法之一——富集评估偏好排序组织法II（Preference Ranking Organization Method for Enrichment Evaluation, PROMETHEE II），对一系列监督式机器学习分类器开展评估。此外，为避免准则权重赋值的任意性，本研究应用了随机多准则可接受性分析（Stochastic Multicriteria Acceptability Analysis, SMAA），通过模拟实验以统计形式生成评估信息。实证测试基于涵盖多组代表性子数据集的知名公开数据集展开，确保样本量存在差异。整体而言，在所采用的MCDA方法与各类数据集的评估场景中，机器学习分类器的评估始终呈现出一系列稳定的排序模式，这为相关研究提供了极具价值的洞见，并给出了具体且实用的解读。最终研究结果提供了稳健支撑，表明当需要同时考量公平性关键指标而非仅关注与预测性能相关的常规维度时，可有效利用MCDA方法对机器学习分类器进行评估。