SEiLLM

Large Language Models (LLMs) are increasinglyutilized in essential customer-facing applications. However, LLMscan exhibit unfair treatment toward different demographicgroups, rooted in underlying stereotypes. These biases may resultin disparities overlooked by conventional methods. Traditionalmethods typically take a narrow perspective of a single demo-graphic group, using limited question structures that overlooksubtler disparities. For example, explicit questions like ’Is a[demographic group] more likely to be [stereotype]?’ fail to capturehow LLMs perpetuate stereotypes in realistic contexts. To addressthese limitations, we introduce a novel method, ‘SEiLLM’, de-signed to systematically quantify disparities in LLMs’ responses.SEiLLM performs a pairwise comparison based on semanticdistances between responses to various prompts, considering theperspectives of different demographic groups simultaneously. Byapplying SEiLLM to multiple prompts that present differentperspectives of the same stereotype, we can effectively capture thesubtle underlying stereotypes and disparities that might otherwisego unnoticed. Our findings reveal that while most LLMs demon-strate progress in mitigating gender biases, significant disparitiespersist for age and ethnicity demographic attributes. Seniorsare consistently more distant from mainstream representationsthan younger demographics, and African-Americans and Asiansexhibit larger disparities compared to other ethnic groups. Theseresults underscore SEiLLM’s effectiveness in identifying latentstereotypes and provide actionable insights for enhancing fairnessin AI systems.