A Qualitative Evaluation of an AI-supported Quiz Application to Assess Learning Progress

Abstract

In a current research project at the Ansbach University of Applied Science, an AI-based quiz function was created to serve as a voluntary student-oriented support offer to determine their learning progress in their respective courses by means of conducting self-assessment quizzes. The application takes lecture scripts as input and applies a question generation model to create questions that students can answer. In order to evaluate the given answers, another language model is involved to perform Natural Language Inference (NLI). Users can engage with the system via a graphical user interface currently provided via a web app. To assess preliminary feasibility and perception of the model prototype, a qualitative focus group discussion following a semi-structured interview guideline prepared by the research team according to similar studies in the education field (Sek et al. 2012) was conducted with five participants. A transcript of the discussion was prepared and analyzed using the qualitative content analysis method according to Kuckartz. Overall, the quiz function was well received by the participants of the focus group. However, the prototype still has potential when it comes to generating meaningful questions and transparently assigning categories to the given answers. Furthermore, the quiz parameters should be individually adjustable by users. In the following paper, the development of the service is illustrated by outlining the considerations for the application design and the training procedure of the language models. Afterwards, the design of the qualitative focus group is described including the presentation of the results.

Mehr zum Titel