Evaluation Tools for Human-AI Interactions Involving Older Adults with Mild Cognitive Impairments

Abstract

As artificial intelligence (AI) systems have already proven useful in human lives generally, there is an opportunity for specialized human-AI interaction (HAI) systems to support and provide care for older adults with mild cognitive impairment (MCI). However, the integration of this technology in this population must be thoughtfully designed to accommodate specific needs and limitations. This includes careful measurement of both humans and systems. We developed an evolving dataset categorizing relevant measurement tools into five groups: cognitive ability, demographics & personality, activity level, state of mind, and perceptions of the AI system. Each instance of the tool being used in the literature cataloged in the dataset is qualified in terms of how likely we would recommend using it in the domain of HAI for older adults with MCI based on contextual factors and internal reliability measures. This dataset will serve as a valuable resource for future research, aiding in the identification of promising areas and trends in AI systems for older adults with MCI as well as providing essential tools for future studies.

Methodology

This dataset was not derived through a typical literature review or survey process, but rather followed a more flexible research method. To collect resources for the dataset, we searched numerous databases to identify studies and review types of publications in journals and conferences between the dates of 2000 to 2022. For the papers that contained extensive reviews of literature or cited original tools, we would further look into the citations of those papers, taking us beyond our limited date range. The tools used were categorized into five groups to broadly distinguish their usage in a study, measuring:

Cognitive ability

Demographics, personality, and experiences

Activity level

State of mind

Perceptions of the AI system

Subsequently, we conducted an examination of their Cronbach’s 𝛼 scores to assess internal reliability. We created tiers based on how likely we would be to recommend using each tool in the domain of human-AI (HAI) with older adults with MCI, as follows:

Tier 1 included tools with Cronbach’s 𝛼 ≥ 0.7 when used with older adults with MCI in experimental settings interacting with AI

Tier 2 included tools with Cronbach’s 𝛼 ≥ 0.7 when used with older adults, with or without MCI, in experimental settings with or without AI interaction

Tier * included tools that satisfy the criteria for Tier 1, but, to the best of our knowledge, lack reported Cronbach’s 𝛼 scores

Tier 3 included all remaining tools that do not meet the criteria for Tier 1, 2, or *

It should be emphasized that a tool may be found in one or more tiers because multiple studies used the same tool yet resulted in varying reliability scores, contexts, etc.

Contribute

Readers are encouraged to reach out to Adam Norton (adam[underscore]norton[at]uml.edu) to recommend additional tools and entries to the dataset.

Publication

This dataset is published as a short contribution to the Human-Robot Interaction (HRI) 2024 conference. The corresponding paper citation is below:

Daisy M. Kiyemba, Jasmin Marwad, Elizabeth J. Carter, and Adam Norton. Evaluation Tools for Human-AI Interactions Involving Older Adults with Mild Cognitive Impairments. In Proceedings of the 2024 ACM/IEEE International Conference on Human-Robot Interaction (HRI ’24), March 11–14, 2024, Boulder, CO, USA. ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3610977.3637474

Acknowledgements

This work was supported by the National Science Foundation (IIS-2112633) as part of the AI-CARING Institute: https://ai-caring.org/