LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction
Abstract
This study addresses the challenge of assessing LLM-as-a-tutor in EFL writing education by integrating pedagogical principles, proposing three key criteria—expert evaluation (quality and characteristics of feedback) and learner assessment (learning outcomes)—to evaluate its real-time essay feedback, thereby laying the groundwork for developing effective LLM tutors tailored to the needs of EFL learners.
Authors: Jieun Han, Haneul Yoo, Junho Myung, Minsun Kim, Hyunseung Lim, Yoonsu Kim, Tak Yeon Lee, Hwajung Hong, Juho Kim, So-Yeon Ahn, Alice Oh
Journal: EMNLP 2024 Workshop, CustomNLP4U
Abstract: In the context of English as a Foreign Language (EFL) writing education, LLM-as-a-tutor can assist students by providing real-time feedback on their essays. However, challenges arise in assessing LLM-as-a-tutor due to differing standards between educational and general use cases. To bridge this gap, we integrate pedagogical principles to assess student-LLM interaction. First, we explore how LLMs can function as English tutors, providing effective essay feedback tailored to students. Second, we propose three criteria to evaluate LLM-as-a-tutor specifically designed for EFL writing education, emphasizing pedagogical aspects. In this process, EFL experts evaluate the feedback from LLM-as-a-tutor regarding (1) quality and (2) characteristics. On the other hand, EFL learners assess their (3) learning outcomes from interaction with LLM-as-a-tutor. This approach lays the groundwork for developing LLMs-as-a-tutor tailored to the needs of EFL learners, advancing the effectiveness of writing education in this context.
Link: Read the full paper
Keywords: Education, LLM, AI, EFL