COFAR
收藏资源简介：
使人类优于现代人工智能 (AI) 模型的一个特征是能够解释超出视觉上显而易见的图像。考虑以下两个自然语言搜索查询- (i) “耐心等待购买冰淇淋的顾客队列” 和 (ii) “前往印度著名的莫卧儿建筑的游客队列”。解释这些查询需要用 (i) 常识来推理，例如将人们解释为顾客或游客，等待购买或去看的行为; 以及 (ii) 与命名视觉实体相关的事实或世界知识，例如，图像中的商店是否出售冰淇淋，或者图像中的地标是否是位于印度的莫卧儿建筑。这种推理不仅仅是视觉识别。为了在图像搜索中实现常识和事实推理，我们提出了一个统一的框架，即知识检索-增强多模态变换器 (KRAMT)，该框架将图像中的命名视觉实体视为通往百科全书知识的门户，并将其与自然语言一起利用查询以获取相关知识。此外，KRAMT无缝集成了视觉内容和基础知识，以学习图像和搜索查询之间的对齐方式。然后，此统一框架用于执行需要常识和事实推理的图像搜索。评估了KRAMT的检索性能，并将其与我们引入的新数据集 (即COFAR) 上的相关方法进行了比较。
One feature that makes humans superior to modern artificial intelligence (AI) models is the ability to interpret images beyond their visually apparent content. Consider the following two natural language search queries: (i) "a queue of customers patiently waiting to buy ice cream" and (ii) "a queue of tourists heading to a famous Mughal architectural landmark in India". Interpreting these queries requires (i) common sense reasoning, such as identifying people as customers or tourists and their behaviors of waiting to make a purchase or visiting a landmark; and (ii) factual or world knowledge associated with named visual entities, for example, whether the shop in the image sells ice cream, or whether the landmark in the image is a Mughal structure located in India. Such reasoning is far beyond mere visual recognition. To enable common sense and factual reasoning in image search, we propose a unified framework, Knowledge Retrieval-Augmented Multimodal Transformer (KRAMT), which treats named visual entities in images as gateways to encyclopedic knowledge, and leverages them together with natural language queries to retrieve relevant knowledge. Furthermore, KRAMT seamlessly integrates visual content and background knowledge to learn the alignment between images and search queries. This unified framework is then used to perform image search tasks that require common sense and factual reasoning. We evaluate the retrieval performance of KRAMT and compare it with related methods on our newly introduced dataset, COFAR.
