WikiMuSA: Wikidata-based Multimodal Semantic data for Art

WikiMuSA: Wikidata-based Multimodal Semantic data for Art WikiMuSA is a multimodal benchmark designed for Multimodal Entity Linking (MEL) and knowledge discovery in the cultural heritage domain. It serves as the official evaluation dataset introduced in the paper: "KARAMEL: Knowledge-Aware Ranking for Multimodal Entity Linking in the Arts via Heterogeneous Graphs" Key Features Explicit & Implicit Mentions: Artworks often depict entities (e.g., architectural ruins, allegories, mythological figures) that are not explicitly named in accompanying text descriptions. Multimodal Alignment: Combines visual depictions (digitized images), natural language descriptions (Wikipedia summaries), and structured knowledge graph connectivity. Heterogeneous Knowledge Graph: Includes a curated Wikidata subgraph containing artistic properties (artist, genre, movement, location) as well as entity-to-entity ontological relationships (e.g., instance of, subclass of, part of, relative, occupation). Multilingual Origin: Derived from multilingual Wikipedia articles translated via NLLB-200 (3.3B) and condensed into visual depiction summaries using LLaMA 3 (8B). Dataset Statistics Unique Artworks: 13,482 Candidate Universe (Depicted Entities): 12,177 Modalities: Vision (RGB Images, 224×224), Text (English summaries), Context (Heterogeneous Knowledge Graph) Splits: 80% Train / 10% Validation / 10% Test (artwork-level disjoint splits to prevent graph leakage) Mention Format: Full article summaries without predefined mention tags/boundaries Archive Contents wikimusa/ ├── annotations.zip │ └── annotations/ │ └── dataset/ │ └── final_annotations/ │ ├── imgs_list.txt # List of associated artwork and entity image files │ └── llama3/ │ ├── train.json # Training split annotations (mention text, targets, metadata) │ ├── val.json # Validation split annotations │ └── test.json # Test split annotations └── graph.zip └── graph/ ├── entities # Node identifiers and types (Artwork, Depicted Object, Class, etc.) ├── links # Typed directed relational edges (depicts, subclass_of, etc.) └── entity_properties.json # Pre-extracted node representations / embeddings Benchmark Comparison Dataset Domain Mentions Entities Mention Type Context Graph Wikidata-MEL General 22,136 17,391 Explicit No MELArt Art 1,616 53,901 Explicit No WikiMuSA (Ours) Art 13,482 12,177 Explicit & Implicit Yes Contact: Raffaele Scaringi ([e-mail]), Department of Computer Science, University of Bari Aldo Moro[cite: 1].