Beyond Labels: Explainable Offensive Meme Detection with LLM-VLM Synergy

The widespread use of memes on social media platforms has significantly increased the propagation of offensive content, necessitatingautomated methods for their detection and interpretation. This study introduces a novel multimodal approach leveraging advancedVision-Language Models (VLMs) and Large Language Models (LLMs) to identify offensive memes, particularly focusing on Hindilanguage content. Our methodology integrates visual embeddings from ImageBind and nomic-embed vision models with textualembeddings generated using multilingual e5-large, mBART50, and mT5 architectures. This unified multimodal framework effectivelyaddresses two hierarchical tasks: detecting offensive memes (Level 1) and distinguishing implicit from explicit offensiveness (Level2). Additionally, we propose a hierarchical zero-shot reasoning architecture that classifies memes and provides explicit reasoningexplaining the nature and context of offensiveness. Comprehensive experiments conducted on a publicly available Hindi meme datasetdemonstrate our model’s superior performance compared to existing state-of-the-art systems in terms of accuracy, interpretability,and robustness. This dual-task approach of detection and nuanced classification, coupled with interpretability, paves the way for moretransparent, accountable, and reliable automated moderation systems on digital platforms.