Perception of speaker sincerity in complex social interactions by cochlear implant users

Understanding insincere language, for example sarcasm and teasing, is a fundamental element of communication and crucial for maintaining social relationships. This can be a challenging task for cochlear implant users (CIs) who receive degraded suprasegmental information important for perceiving a speaker’s attitude. We measured perception of speaker sincerity (literal positive, literal negative, sarcasm, and teasing) in 16 adults with CIs using an established video inventory. Participants were presented with audio-only and audio-visual social interactions between two people with and without supporting verbal context. They were instructed to describe the content of the conversation and answer whether the speakers meant what they said. Results showed that subjects could not always identify speaker sincerity, even when the content of the conversation was perfectly understood. This deficit was greater for perceiving insincere relative to sincere utterances. Performance improved when additional visual cues or verbal context cues were provided. Subjects who were better at perceiving the content of the interactions in the audio-only condition benefited more from having additional visual cues for judging the speaker’s sincerity, suggesting that the two modalities compete for cognitive recourses. Percentage scores for understanding the content also did not correlate with that for extracting speaker sincerity, suggesting that what was said vs. how it was said were perceived using unrelated segmental versus suprasegmental cues. Our results further showed that subjects who had access to lower-order resolved harmonic information provided by a hearing aids in the contralateral ear identified speaker sincerity better than those who used implants alone. The results suggest that measuring speech recognition alone in CI users do not fully describe the outcome and our findings stress the importance of measuring social communication functions in people with CIs.

理解非真诚性语言（如讽刺与打趣）是人际沟通的核心要素，对维系社会关系至关重要。对于人工耳蜗植入者（Cochlear Implant Users, CIs）而言，这一任务极具挑战性——这类人群接收的、用于感知说话者态度的超音段信息（suprasegmental information）存在缺损。
我们采用经过验证的视频测试集，对16名成年人工耳蜗植入者的说话者真诚度感知能力（包括字面正向、字面负向、讽刺与打趣四类表达）进行了测评。实验中，向被试展示了两类社交互动场景：仅音频形式，以及音视频结合形式，且每种场景均包含或不包含辅助言语语境。要求被试先描述对话内容，再判断说话者是否言为心声。
实验结果显示，即便被试完全理解了对话内容，也并非总能准确识别说话者的真诚度。相较于真诚话语，被试对非真诚话语的感知缺陷更为显著。当添加视觉线索或言语语境线索时，被试的测评表现会有所提升。在仅音频条件下，更擅长识别互动内容的被试，从额外视觉线索中获得的真诚度判断增益更为明显，这表明两种感知通道存在认知资源（cognitive resources）竞争。对话内容理解的得分与说话者真诚度提取的得分之间并无相关性，这表明，对“说了什么”的感知依赖于音段线索（segmental cues），而对“如何说”的感知则依赖于超音段信息，二者的感知线索互不相关。我们的研究还发现，能够利用对侧耳助听器（hearing aid in the contralateral ear）提供的低阶解析谐波信息的被试，比仅使用人工耳蜗植入体的被试更能准确识别说话者的真诚度。上述结果表明，仅通过言语识别测评无法全面反映人工耳蜗植入者的康复效果，本研究结果也强调了对人工耳蜗植入者开展社交沟通功能测评的重要性。