TruthEval

TruthEval数据集是由滑铁卢大学计算机科学学院创建，旨在评估大型语言模型（LLM）的真实性和可靠性。该数据集包含885条关于敏感话题的陈述，分为六个类别：事实、阴谋论、争议、误解、刻板印象和虚构。这些陈述经过手工筛选，包含已知真实值，用于区分LLM的能力和随机性。数据集的创建过程涉及从维基百科、GPT-3等来源收集陈述，并通过语义去重技术进行精炼。TruthEval数据集主要用于评估LLM在理解世界知识方面的能力，特别是在处理具有明确真实值和模糊真实值的话题时的表现。

The TruthEval dataset was developed by the School of Computer Science at the University of Waterloo, with the core objective of evaluating the authenticity and reliability of Large Language Models (LLMs). It contains 885 statements related to sensitive topics, which are categorized into six groups: factual content, conspiracy theories, controversial topics, misconceptions, stereotypes, and fictional works. All these statements are manually curated with known ground truth labels, designed to distinguish between the genuine capabilities of LLMs and random responses. The construction process of the dataset entails collecting statements from sources including Wikipedia, GPT-3 and other resources, followed by refinement via semantic deduplication techniques. The TruthEval dataset is primarily utilized to assess LLMs' proficiency in comprehending world knowledge, especially their performance when handling topics with both explicit and ambiguous ground truth values.