Exploring AI in Education: A Quantitative Study of a Service-Oriented University Chatbot

Abstract

The following paper presents the evaluation of an artificially intelligent assistant system (DIAS) with a service-oriented chatbot as a central communication element. The conversational AI (Artificial Intelligence) is supposed to increase information transparency in higher education environments and thus support students, teachers, and administrative staff. The exploratory study had two objectives: first, we intended to find out about the usability and utility of the DIAS chatbot using the CUQ (Chatbot Usability Questionnaire) score and benchmark the results against other conversational agents. Secondly, we were interested in possible effects among the different variables of interest, which could contribute to further theory development of chatbots in education. The results show that the DIAS chatbot scored above average, and can support students in finding relevant information, particularly if they use the assistant frequently. Positive aspects included the intuitive use, a welcoming persona (expressed in design & language) and easy navigation. The negative feedback showed potential for improvement particularly in content quality and handling dialogue mistakes, which is a general shortcoming of conversational AI at this development stage. The results can be used as a guidance for future research and theory building. However, they must be considered carefully due to several study limitations.

Mehr zum Titel