Cross-Modal Correspondences Enhance Performance on a Colour-to-Sound Sensory Substitution Device

Visual sensory substitution devices (SSDs) can represent visual characteristics through distinct patterns of sound, allowing a visually impaired user access to visual information. Previous SSDs have avoided colour and when they do encode colour, have assigned sounds to colour in a largely unprincipled way. This study introduces a new tablet-based SSD termed the ‘Creole’ (so called because it combines tactile scanning with image sonification) and a new algorithm for converting colour to sound that is based on established cross-modal correspondences (intuitive mappings between different sensory dimensions). To test the utility of correspondences, we examined the colour–sound associative memory and object recognition abilities of sighted users who had their device either coded in line with or opposite to sound–colour correspondences. Improved colour memory and reduced colour-errors were made by users who had the correspondence-based mappings. Interestingly, the colour–sound mappings that provided the highest improvements during the associative memory task also saw the greatest gains for recognising realistic objects that also featured these colours, indicating a transfer of abilities from memory to recognition. These users were also marginally better at matching sounds to images varying in luminance, even though luminance was coded identically across the different versions of the device. These findings are discussed with relevance for both colour and correspondences for sensory substitution use.

视觉感官替代设备（SSDs）能够通过独特的声音模式来表征视觉特征，从而使得视障用户能够获取视觉信息。以往的研究中，SSDs通常避免对颜色进行编码，而当它们确实进行编码时，对颜色的声音分配也大多缺乏系统性。本研究引入了一种新的基于平板的SSD，称为‘克里奥尔’（之所以得名，是因为它结合了触觉扫描和图像声化），以及一种将颜色转换为声音的新算法，该算法基于已建立的跨模态对应关系（不同感官维度之间的直观映射）。为了检验对应关系的有效性，我们研究了具有视觉的受试者在设备编码遵循或违反声音-颜色对应关系时的颜色-声音联想记忆和物体识别能力。采用基于对应关系的映射的用户在颜色记忆方面取得了改善，并减少了颜色错误。有趣的是，在联想记忆任务中提供最大改进的颜色-声音映射，在识别具有这些颜色的真实物体时也取得了最大的收益，这表明了从记忆到识别能力的迁移。这些用户在将声音与不同亮度变化图像匹配方面也略有优势，尽管亮度在设备的不同版本中编码方式相同。这些发现与感官替代使用中的颜色和对应关系的相关性进行了讨论。