Dynamic Knowledge benchmark (DyKnow)

DyKnow是一个动态知识基准，旨在评估大型语言模型（LLMs）中时间敏感知识的更新情况。该数据集包含政治、体育和组织等领域的实时数据，通过从Wikidata知识库中实时检索每个目标事实问题的最新答案来构建。DyKnow不仅用于评估LLMs的时效性，还可通过共享代码和脚本扩展到其他领域。数据集的构建涉及选择特定领域的实时事实，并通过手动控制确保事实的准确性和变化频率。DyKnow is a dynamic knowledge benchmark designed to evaluate the temporal knowledge updating capabilities of Large Language Models (LLMs). This dataset comprises real-time data across domains such as politics, sports and organizations, and is constructed by retrieving the latest answers to each target factual question in real time from the Wikidata knowledge base. Beyond evaluating the timeliness of LLMs, DyKnow can also be extended to other domains via shared code and scripts. The construction of this dataset involves selecting real-time facts from specific domains, and ensuring the accuracy and update frequency of these facts through manual oversight.

搜集汇总

背景与挑战

背景概述

DyKnow是一个动态知识基准，专注于评估大型语言模型在时间敏感知识上的更新能力。它包含政治、体育和组织等领域的实时数据，通过从Wikidata知识库实时检索最新答案构建，并支持扩展到其他领域，以促进模型时效性研究。