ForceMapping: learning visual-force features from vision for soft objects manipulation

To realize delicate manipulation of soft/rigid objects, incorporating some notion of force into the controller is essential. Such forces can either be measured explicitly via onboard sensors or estimated from visual input through an offline-trained network. For visuomotor control, unlike state-based models, visual input (i.e. RGB images) is first encoded into a low-dimensional latent before feeding into some low-level policy. The encoder network is usually trained end-to-end using supervised policy loss or pre-trained with unsupervised RGB reconstruction loss. This work proposes a learning approach that implicitly integrates force-relevant features into the latent image encoding, rather than explicit force integration. Our proposal, ForceMapping, incorporates a supervised force prediction loss as an auxiliary optimization objective alongside unsupervised RGB reconstruction loss or policy loss. For soft bodies, forces are estimated from the raw RGB image input, given the correlation between object deformation and imposed forces. To validate our approach, we designed a soft object stacking task requiring force-aware manipulation. Soft-body deformation was efficiently rendered through a fast and stable pipeline, utilizing states obtained from rigid-body simulation. Our experiments demonstrated that ForceMapping encouraged attention to and encoding of force-relevant features, leading to increased task success rates (from 25% to 75%) and reduced deviations from a target critical load (from 300% to 40%) compared to only-RGB model.