CompAct Dataset for Sequential Compositional Generalization in Multimodal Models

CompAct (Compositional Activities) presents a comprehensive benchmark for assessing the compositional generalization abilities of Sequential Multimodal Models. CompAct is a carefully constructed, perceptually grounded dataset set within a rich backdrop of egocentric kitchen activity videos. Each instance in our dataset is represented with a combination of raw video footage, naturally occurring sound, and crowd-sourced step-by-step descriptions. More importantly, our setup ensures that the individual concepts are consistently distributed across training and evaluation sets, while their compositions are novel in the evaluation set. We conduct a comprehensive assessment of several unimodal and multimodal models.