Development of compositionality through interactive learning of language and action of robots

Humans excel at applying learned behavior to unlearned situations. A crucial component of this generalization behavior is our ability to compose/decompose a whole into reusable parts, an attribute known as compositionality. One of the fundamental questions in robotics concerns this characteristic. \"How can linguistic compositionality be developed concomitantly with sensorimotor skills through associative learning, particularly when individuals only learn partial linguistic compositions and their corresponding sensorimotor patterns?\" To address this question, we propose a brain-inspired neural network model that integrates vision, proprioception, and language into a framework of predictive coding and active inference, based on the free-energy principle. The effectiveness and capabilities of this model were assessed through various simulation experiments conducted with a robot arm. Our results show that generalization in learning to unlearned verb-noun compositions, is significantly enhan..., The data was collected with a robotic arm (called Torobo Arm) and an external RGB camera. The vision data was preprocessed with openCV in python. More details can be found in the github repository https://github.com/oist-cnru/FEP-based-model-of-Embodied-Language. , , # Development of compositionality through interactive learning of language and action of robots

[https://doi.org/10.5061/dryad.xsj3tx9qc](https://doi.org/10.5061/dryad.xsj3tx9qc)

The dataset used for training and evaluating the model referenced in the article titled \"Development of compositionality through interactive learning of language and action of robots\". The dataset was collected from robotic arm (Torobo Arm) and an external RGB camera.

## Description of the data and file structure

The dataset contains individual h5py files of synchronized visuo-proprioceptive data. Each image is 64x64x3 vector and joint angles used for proprioception data are 60 dimensional vectors. The dataset was preprocessed using openCV python for images. The dataset with synchronized visuo-proprioceptive sequences and language were created with 'h5py' python  library. Each file in the dataset also contains the language corresponding to the action performed by the robot. The language is provided in one-h...