On the Impact of Self-efficacy on Assessment of User Experience in Customer Service Chatbot Conversations

Yuexin Cao; Vicente Ivan Sanchez Carmona; Xiaoyi Liu; Changjian Hu; Neslihan Iskender; André Beyer; Sebastian Möller; Tim Polzehl

In: IWSDS 2021. International Workshop on Spoken Dialogue Systems Technology (IWSDS-2021), Springer, 2021.


In this paper, we analyze influencing factors for the assessment of user experience (UX) from a chatbot operating in the domain of technical customer support. To find out which UX factors can be assessed reliably in a crowdsourcing setup, we conduct a crowd-based UX assessment study through a set of scenario-based tasks and analyze the UX assessments in the light of influencing user characteristics, i.e., self-reported self-efficacy of individual users. By segmenting users according to self-efficacy, we find significant differences in UX assessment and expectations of users with respect to a series of UX constituents like acceptability, task efficiency, system error, ease of use, naturalness, personality and promoter score. Our results strongly suggest a potential application for essential personalization and user adaptation strategies utilizing self-efficacy for the personalization of technical customer support chatbots. Therefore, we recommend considering its influence when designing chatbot adaptation strategies for maximized customer experience.

Deutsches Forschungszentrum für Künstliche Intelligenz
German Research Center for Artificial Intelligence