Role of SMMs in AI-advised Decision Support

A crucial requirement for effective human-AI teaming is the ability to form mutually compatible mental models that help humans discern when the AI can be relied upon and how to best complement each other. To understand the impact of having this mutual understanding, we conducted an experiment where participants (N=125), tasked as disaster relief planners, were assisted by an AI agent that recommends optimal allocation of a resource based on several information sources. The interaction effects of mental models between the human and the AI were studied. Our analysis indicates that accurate mutual understanding between the human-AI dyad can compensate for some gaps in task understanding. However, one-way mental models between the user and the AI induce a trade-off space between accuracy, task speed, and user effort. In teams where only the user has a mental model of the AI, performance improvement was accompanied by increased task load (active information access) and subsequent erosion in users’ perception of the AI. Conversely, users paired with an AI possessing a model of the user showed reductions in instances of active information access, but these reductions came at the cost of complacency and blind compliance by the user. These findings motivate a shift towards designing AI systems that can support the user by providing proactive support, while simultaneously providing clear guidelines for the user to better understand the AI teammate they are working with. These results also serve as evidence for the utility of shared mental models in human-AI teams.