Smooth Talking: Generative Versus Scripted Coaching Chatbot Adoption and Efficacy Comparison

Workplace coaching is an essential individual and organizational development intervention, but high costs limit its application. AI coaching chatbots could scale coaching, however traditional scripted chatbots lacks mainstream adoption due to limited context awareness and robotic responses. Recent advances in large language models and generative AI offer an alternative, however generative AI is linked to risks and controversies. Should coaching chatbots use generative AI? We compared a scripted, rules-based coaching chatbot (ScriptBot, n=126) with a generative AI chatbot (GenBot, n=116) in a randomized controlled trail experiment. We measured technology adoption (UTAUT2), self-determination (autonomy, competence, relatedness) and goal attainment. For all technology adoption and self-determination variables, GenBot outperformed ScriptBot. However; for goal attainment there was no significant difference. Using generative AI in coaching chatbots improves adoption and coaching outcomes, but for highly structured interactions such as goal attainment, scripted conversations (with lower risk) seem to work just as well.