Three-dimensional shape Cues Affect Human and Artificial Recognition Systems Differently (Dataset)

Humans and neural networks use shape and texture information differently. While humans weigh shape heavily in their ultimate classification decisions, neural networks are more biased toward texture cues. Many tests of shape vs. texture bias have focused on shape recognition from an object’s external contour. However, shape information is also conveyed through internal contours, shading, and attached shadows, especially when an object is viewed from noncanonical perspectives. Using models from ShapeNet, we created datasets of 120,000 texture-substituted images of objects from many viewpoints, with and without shading and attached shadows. We tested humans’ and several neural networks’ ability to classify these objects by both their shape and their texture. Humans were much better at classifying texture-substituted objects by their shape than any network, although these differences were greater when shape was defined only by the external contour than when 3D cues were included. Our findings suggest that networks’ texture bias is reduced when 3D cues are included in images. We next tested whether the inclusion of 3D cues benefited humans and neural networks more for images of objects viewed from canonical or noncanonical perspectives. Consistent with earlier research, we found that 3D cues primarily benefited humans for noncanonical images. For neural networks, the greatest performance gains were for canonical images. These findings suggest fundamental differences in how humans and networks use shading and attached shadows for object recognition. We argue that humans use these cues to infer objects’ 3D structures, while neural networks use them as another surface-level cue, like texture. This dataset contains 2D and 3D images, MATLAB and Python scripts/notebooks, and anonymized human and neural network data.