Towards efficient human-machine collaboration: Effects of gaze-driven feedback and engagement on performance

Can listener gaze facilitate goal-oriented human-machine collaboration? In order to jointly solve a task, interlocutors need to establish reference in a shared environment. Our usual mean of communication is natural language but it is not interpreted in isolation. Other modalities (e.g. gaze and gestures) support communicative success in situated interactions. Specifically, an artificial speaker (natural language generation NLG system) can exploit listener gaze to realise an effective interaction strategy by responding to it with verbal feedback. Considering an object identification task for assembly under system instruction, we show that proactive feedback generated on the basis of object inspections can improve task performance. In particular, providing information incrementally in subsequent chunks turns out to be more efficient than giving the description in one piece. Moreover, feedback’s informativity not only leads to more efficient interactions but also influences the overall expectation to the capabilities of the NLG system. This expectation determines to what extend the listener wants to cooperate and will engage with the NLG system. The more intense listeners engage with the system, the more effective is the information uptake and the better the task performance even when the system’s responses are less informative.