PIGLeT-Vis

This is the PIGLeT-Vis dataset introduced in the EACL Findings 2023 paper titled "Learning the Effects of Physical Actions in a Multi-modal Environment".

The dataset contains images, symbolic descriptions, and annotated actions that describe a simulated AI2THOR environment before and after an action is taken. The task is to predict the effects of actions given the pre-conditions.

The original dataset PIGLeT was created by Zellers et. al. 2021 (ACL 2021). PIGLeT-Vis filters and cleans the original dataset for it to be used to evaluate physical commonsense in a multi-modal setting (using images instead of symbolic representations as inputs).

ABSTRACT: Large Language Models (LLMs) handle physical commonsense information inadequately. As a result of being trained in a disembodied setting, LLMs often fail to predict an action’s outcome in a given environment. However, predicting the effects of an action before it is executed is crucial in planning, where coherent sequences of actions are often needed to achieve a goal. Therefore, we introduce the multi-modal task of predicting the outcomes of actions solely from realistic sensory inputs (images and text). Next, we extend an LLM to model latent representations of objects to better predict action outcomes in an environment. We show that multi-modal models can capture physical commonsense when augmented with visual information. Finally, we evaluate our model’s performance on novel actions and objects and find that combining modalities help models to generalize and learn physical commonsense reasoning better.