Learning in the digital world requires not only technological skills for using digital tools but also ethical skills to critically reflect on (in)adequate digital media use and potential negative consequences. These skills are particularly crucial in professions dealing with public welfare and societal issues. Inter alia, those are teachers who educate the youth, legal professionals who judge (il)legal behavior regarding media, or computer scientists who bear responsibility when developing algorithms. Accordingly, higher education students studying teacher education, law studies, or computer science studies should develop ethical skills for the digital world. This study examined how higher education students perceive problematic media behaviors and which digital competences they regard relevant for ethical issues. In addition, the study investigated whether students of teacher education, law studies, and computer science studies differ in their perceptions. To this end, an online survey with N = 461 participating students was conducted. Study results indicated that higher education students perceived problematic media behaviors as such with posting inappropriate content identified as the most problematic. Furthermore, students considered several digital competences as relevant for ethical issues with protecting and acting safely as most relevant. In-depth analyses uncovered subject-specific differences with computer science students being most ethically savvy, albeit differences were only of small effect size. The study provides valuable insights into the intersection of digital competences, ethical considerations, and academic disciplines. In the future, longitudinal and training studies will help to understand how differences emerge and whether students of different study subjects benefit from digital ethics training.

Does the AI agree? Inter-rater agreement in learning diary evaluation

Theory

Learning diaries as formative assessments are promising to support the learning process and stimulate reflection. In an open learning diary, learners apply learning strategies to reflect on their learning and deepen their knowledge. To guide learners, learning diaries can be structured according to different learning strategies. However, grading and feedback on learning diaries is effortful for teachers. Artificial intelligence (AI) may assist teachers in the evaluation process. A prerequisite is that AI and teachers show high inter-rater agreement.

Research Questions

The current study aimed to analyze the agreement between teachers and ChatGPT-4o by examining four separately assessed learning strategy categories of a learning diary in adult education (organization, in-depth elaboration, transfer-supporting elaboration, and metacognition). The two research questions were:

• Q1: How accurate is the overall agreement between teachers and ChatGPT-4o, and are there differences across different learning strategy categories?

• Q2: Does the inter-rater agreement differ between teachers across the four learning strategy categories?

Method

Seven different adult education teachers and ChatGPT-4o evaluated a total of 540 learning diary entries. Each teacher assessed approximately 65 entries. Teachers were trained in criteria-based evaluation per learning strategy category. An engineered prompt supported the ChatGPT-4o model.

Teacher ratings served as the reference for the inter-rater agreement. Absolute accuracy and under-/overestimation (bias) were calculated for each learning strategy category as accuracy measures. Furthermore, overall accuracy values were calculated across the four categories for absolute accuracy and bias.

A doubly multivariate repeated measures ANOVA was conducted with the four learning strategy categories as repeated measures and absolute accuracy and bias as measures to test for accuracy differences between the learning strategy categories (Q1). Additionally, teacher was used as a between-subjects factor. Thus, the interaction of the learning strategy category and rating teacher regarding accuracy could be tested statistically (Q2).

Künstliche Intelligenz (KI) kann im Prozess der Leistungsbewertung assistieren und diesen transformieren. Besonders lohnend scheint eine KI-Assistenz bei der Bewertung von komplexem, geschriebenem Text. Jedoch ist der Einsatz von KI im Bewertungsprozess «hochriskant» (EU 2024) und bedarf umfangreicher Analysen. Die vorliegende Studie untersucht, inwiefern ChatGPT-4o die Auswertung und Interpretation von Lerntagebucheinträgen objektiv vornehmen kann. Dafür werden 757 Lerntagebucheinträge aus der geförderten Weiterbildung in Deutschland von Mensch und Maschine bewertet. Sowohl Mensch als auch Maschine erhalten hierzu Kriterien, nach denen die Bewertung vorzunehmen ist; ChatGPT-4o wird diesbezüglich mit einem Prompt unterstützt. Die Übereinstimmung der Bewertungen wird anhand der Masse Sensitivität und Spezifität gemessen. Die Ergebnisse zeigen, dass die Bewertungsvorschläge von ChatGPT-4o eine moderate bis hohe Übereinstimmung mit den menschlichen Bewertungen aufweisen; gleichzeitig neigt ChatGPT-4o jedoch zu einer optimistischen Bewertung der Lerntagebucheinträge. Die Ergebnisse weisen darauf hin, dass eine hybride Intelligenz, also eine Kombination der Stärken von Mensch und Maschine, gewinnbringend für Bewertungsprozesse sein kann. Künftig denkbar sind halbautomatisierte Bewertungsprozesse von Lerntagebucheinträgen, in denen die KI die Bewertung der Lerntagebucheinträge übernimmt und Lehrkräfte bei kritischen Fällen regulierend eingreifen. So könnte die Korrektureffizienz ohne bedeutende Qualitätsverluste gesteigert werden.

Learning diaries are reflective tools, often used as formative assessments in adult education with the aim to promote cognitive and metacognitive learning strategies. As grading of and feedback on learning diaries is effortful for teachers, artificial intelligence (AI) may assist teachers in evaluating learning diaries. A prerequisite is that AI's ratings show high accordance with the teachers' ratings. AI accuracy, measured via absolute accuracy and bias, is the focus of the current study with N = 540 learning diary entries focusing on learning strategies, seven teachers, and ChatGPT-4o. Findings revealed that AI evaluations align closely with teacher assessments, indicated by high overall accuracy and low bias. Interestingly, the accuracy varied based on the types of learning strategies assessed in the diaries. Additionally, individual teacher assessments influenced the alignment between human and AI evaluations, suggesting that teachers applied their profession-specific expertise to the assessment process while AI produced somewhat generic evaluations. Overall, the study results indicate that AI can enhance the efficiency of formative assessments while providing timely feedback to learners.

For successful learning, monitoring one's learning behavior and outcomes is essential. While both the use of monitoring strategies and monitoring judgment accuracy are regarded as indicators for metacognitive monitoring, their joint contribution to metacognitive monitoring and their interrelationship is unclear. The current study investigated how monitoring strategies and monitoring judgments (spontaneous as well as prompted judgments) change micro-genetically and influence each other reciprocally during an exam preparation phase. In this diary study, N = 273 higher education students completed knowledge tests and surveys on five consecutive days. Results indicated interindividual differences in dependence of prior performance and gender for the accuracy of prompted judgments but not for monitoring strategies or spontaneous judgments. Random intercept cross-lagged panel models revealed that higher quality monitoring strategies led to the spontaneous formation of monitoring judgments, while the quality of monitoring strategies and prompted judgment accuracy did not influence each other significantly.

Evidence syntheses, such as systematic reviews, aim to summarise the current state of research in a field, often using the publication language of a study as a criterion for inclusion or exclusion. However, this has serious implications for capturing evidence from a wider range of geographical areas, and the potential for linguistic bias. In order to explore this issue, a trilingual tertiary mapping review of 446 evidence syntheses within the field of educational technology (EdTech) and published in English, Spanish and German was undertaken, analysing the frequency of multi- and monolingual evidence syntheses, reasons for language choice by research teams, and the composition of research teams in multi- and monolingual evidence syntheses. Items were included if they were a form of evidence synthesis with an explicit method section, indexed within ERIC, Scopus, Web of Science, Dialnet, FIS-Bildung, or Google Scholar, education-related, and published between 1983 and May 2022. The results showed that only eight languages were considered in published syntheses, only five languages were used to construct search strings, most evidence syntheses included research published in English without explaining why, and multilingual research team composition did not predict multilingual evidence syntheses. The findings suggest the need to address publication languages not only as a formal criterion but as an integral aspect of methodological approach, influencing the content and scope of syntheses in educational research.

Synchronous online learning is situated in synchronous webinars, virtual classrooms, and videoconferences, in which students and teachers communicate live over the Internet across distant geographical locations. Shared virtual platforms with voice-over-IP technology and webcams are used to interact ubiquitously and synchronously in real-time. During the COVID-19 pandemic, the use of synchronous online learning environments in primary, secondary, higher, and continuous education has grown massively. In this chapter, we aim to identify design recommendations that help increase the effectiveness of synchronous online learning. First, we provide an overview of recent meta-analyses on the effectiveness of synchronous online learning compared to synchronous face-to-face and asynchronous online learning. Second, we use this empirical evidence to develop design principles that can guide teachers, trainers, and educators in designing effective synchronous online learning environments. Third, we discuss a research agenda for future studies on synchronous online learning, webinars, and videoconferences in education, and we reflect on some practical affordances, including the use of webcams and breakout rooms.

Learning from texts means acquiring and applying knowledge, which requires students to judge their text comprehension accurately. However, students usually overestimate their comprehension, which can be caused by a misalignment between the cues used to judge one’s comprehension and the cognitive requirements of future test questions. Therefore, reading instructions might help students to use more valid cues and hence to make more accurate judgments. In two randomized experiments, we investigated the effect of application instructions (in contrast to general and memory instructions) on judgment bias regarding memory test performance and application test performance. In Experiment 1, 131 pre-service teacher students read two texts: For the first text (pretest phase), all participants received general reading instructions. For the second text (testing phase), they received one of the three reading instructions. Main results were that the general reading instructions in the pretest phase resulted in underestimation for memory test performance and overestimation for application test performance. Results from the testing phase yielded mixed effects and, overall, no strong evidence that reading instructions, and in particular application instructions, are beneficial for debiasing judgments of comprehension. Experiment 2 (N = 164 pre-service teachers) restudied the effects with the same texts but a different study design. Results replicated the effects found in the testing phase of Experiment 1. Overall, the results indicated that reading instructions without further support are not sufficient to help students to accurately judge their comprehension and suggested that text characteristics might impact the effect of reading instructions on judgment bias.