Development of compositionality through interactive learning of language and action of robots

Humans excel at applying learned behavior to unlearned situations. A
crucial component of this generalization behavior is our ability to
compose/decompose a whole into reusable parts, an attribute known as
compositionality. One of the fundamental questions in robotics concerns
this characteristic. "How can linguistic compositionality be
developed concomitantly with sensorimotor skills through associative
learning, particularly when individuals only learn partial linguistic
compositions and their corresponding sensorimotor patterns?" To
address this question, we propose a brain-inspired neural network model
that integrates vision, proprioception, and language into a framework of
predictive coding and active inference, based on the free-energy
principle. The effectiveness and capabilities of this model were assessed
through various simulation experiments conducted with a robot arm. Our
results show that generalization in learning to unlearned verb-noun
compositions, is significantly enhanced when training variations of task
composition are increased. We attribute this to self-organized
compositional structures in linguistic latent state space being influenced
significantly by sensorimotor learning. Ablation studies show that visual
attention and working memory are essential to accurately generate
visuo-motor sequences to achieve linguistically represented goals. These
insights advance our understanding of mechanisms underlying development of
compositionality through interactions of linguistic and sensorimotor
experience.