Bias in Open-Ended Language Generation Dataset (BOLD)

BOLD数据集是由亚马逊Alexa人工智能-NU美国团队创建的大规模英语文本生成偏见基准数据集，包含23,679个用于偏见基准测试的英语文本生成提示，涵盖职业、性别、种族、宗教和政治意识形态五个领域。数据集的提示从英文维基百科文章中提取，代表了来自不同作者的自然发生文本。BOLD旨在系统地研究和基准化开放式语言生成中的社会偏见，并提出了新的自动化度量标准，如毒性、心理语言学规范和文本性别极性，以从多个角度测量社会偏见。数据集的应用领域包括对话机器人、自动故事讲述等下游任务，旨在解决语言模型中嵌入的偏见问题。

The BOLD dataset is a large-scale benchmark dataset for text generation bias, created by the Amazon Alexa AI-NU US Team. It contains 23,679 English text generation prompts for bias benchmarking, covering five domains: occupation, gender, race, religion, and political ideology. The prompts in the dataset are extracted from English Wikipedia articles, representing naturally occurring texts from various authors. BOLD aims to systematically study and benchmark social biases in open-ended language generation, and proposes novel automated metrics including toxicity, psycholinguistic norms, and text gender polarity to measure social biases from multiple perspectives. The dataset's application scenarios include downstream tasks such as conversational robots and automatic storytelling, with the goal of addressing the bias issue embedded in language models.