CounterEval: Towards Unifying Evaluation of Counterfactual Explanations

CounterEval offers a human-evaluated dataset of 30 counterfactual scenarios, engineered to serve as a benchmark for evaluating a wide range of counterfactual explanation generation frameworks. Each scenario has been strategically designed to exhibit varying levels across multiple explanatory quality metrics, such as Feasibility, Consistency, Trust, Completeness, Fairness, and Complexity, while maintaining high Understandability and using Overall Satisfaction as a general measure of quality. The scenarios are based on well-known datasets like the Adult dataset and the Pima Indians Diabetes dataset, which include a mix of categorical and continuous data. Additional fabricated scenarios were created to explore extreme values in certain metrics. Each scenario has been evaluated by 206 individuals recruited through the Prolific platform, across all metrics, enabling detailed analysis of how individual explanatory metrics influence the overall satisfaction score.