CUBE-MT: A Cultural Benchmark for Multimodal Knowledge Graph Construction with Generative AI

Cultural heritage institutions (GLAM) have a societal role of guaranteeing access to their collections independently of location or background, as a pledge for knowledge equity. However, around one billion people (15% of the world’s population) experience some form of disability that hinders such access, especially when considering the multimedia gap: differences in the use of different types of media to convey content to different audiences. Knowledge Graphs are increasingly becoming more multimodal by supporting images and text. However, the extent to which they address the multimedia gap for people with disabilities is not well understood due to the lack of appropriate cultural evaluation frameworks. To address this, here we propose CUBE-MT, a benchmark and dataset that leverages generative AI to build Multimodal Knowledge Graphs (MMKGs) with surrogate, multimedia representations that adapt to the sensory capacities of cultural heritage collection users. We extend the CUBE (CUltural BEnchmark for Text-to-Image models) benchmark and provide 7 modalities (text, images, Braille, speech, music, and 3D models); a collection of prompts to account for their cultural awareness; and a dataset with the resulting MMKG linked to Wikidata. We show usage and evaluate the effectiveness of our approach in an expert survey and a user study with people with aphasia, focusing on perceptual and comprehension differences between original MMKGS objects and those generated by AI.