Prioritizing Metamorphic Relations for Bias Detection

This research hypothesizes that sentence diversity metrics can enhance the prioritization of metamorphic relations (MRs) for fairness testing in Large Language Models (LLMs) such as GPT 4.0 and LLaMA 3.0. The goal is to improve fault detection rates and reduce time to first failure (TFF) compared to existing methods like random, distance-based, and fault-based ordering. To test this, 4,700 test cases were generated from templates containing placeholders for sensitive attributes, which were systematically modified using various MRs. The responses from LLMs were analyzed using diversity metrics, including cosine similarity (embedding-level similarity), lexical diversity (vocabulary variation), Named Entity Recognition (NER) diversity (changes in named entities), semantic similarity (using SentenceTransformer embeddings), sentiment similarity, and tone diversity (emotional tone consistency). The findings reveal that diversity-based prioritization significantly outperforms existing methods. It achieved higher fault detection rates and reduced TFF, allowing quicker and more effective identification of fairness faults. Among the metrics, tone diversity detected the highest number of fairness bugs, highlighting its utility in uncovering biases related to emotional tone. NER diversity effectively identified biases linked to named entities, while semantic and sentiment similarity captured more nuanced fairness violations. Additionally, intersectional biases—arising from combinations of sensitive attributes such as religion, political views, and economic status—frequently revealed fairness issues, emphasizing the need for targeted intersectional analysis. The results demonstrate that integrating sentence diversity metrics into MR prioritization provides a more efficient and comprehensive approach to fairness testing. By reducing the time required to identify faults and improving test coverage, this methodology can enhance fairness evaluation in high-stakes applications such as healthcare, finance, and education. Furthermore, the scalability of this approach offers a generalized framework for testing other AI systems, contributing to the development of more equitable and robust AI technologies.本研究提出假设：句子多样性指标可优化大语言模型（Large Language Models, LLMs）如GPT 4.0、LLaMA 3.0等的公平性测试中蜕变关系（metamorphic relations, MRs）的优先级排序。相较于随机排序、基于距离的排序以及基于故障的排序等现有方法，本研究目标在于提升故障检出率并缩短首次故障出现时间（time to first failure, TFF）。为验证该假设，研究人员从包含敏感属性占位符的模板中生成了4700个测试用例，并通过各类蜕变关系对测试用例进行系统性修改。随后，研究采用多种多样性指标对大语言模型的输出响应展开分析，包括余弦相似度（嵌入级相似度）、词汇多样性（词汇变异程度）、命名实体识别（Named Entity Recognition, NER）多样性（命名实体变化情况）、语义相似度（基于SentenceTransformer嵌入）、情感相似度以及语气多样性（情感语气一致性）。 研究结果显示，基于多样性的优先级排序方法显著优于现有方法：其可实现更高的故障检出率并缩短首次故障出现时间，能够更快且更高效地识别公平性故障。在各类指标中，语气多样性检出的公平性漏洞数量最多，凸显其在挖掘与情感语气相关的偏见方面的应用价值；命名实体识别多样性可有效识别与命名实体相关的偏见，而语义相似度与情感相似度则能捕捉更为细微的公平性违规行为。此外，交叉性偏见——由宗教、政治观点、经济地位等多种敏感属性组合引发——常暴露公平性问题，这凸显了开展针对性交叉性分析的必要性。 研究结果表明，将句子多样性指标融入蜕变关系优先级排序，可为公平性测试提供更高效、更全面的解决方案。通过缩短故障识别耗时并提升测试覆盖度，该方法可强化医疗、金融、教育等高风险应用场景中的公平性评估。此外，该方法具备可扩展性，可为测试其他人工智能系统提供通用框架，有助于推动更公平且更稳健的人工智能技术发展。