Classifying the Concept Image in Student Responses with Large Language Models

Abstract

In big lectures, it can be difficult to maintain an extensive overview of the learners’ state of knowledge. In this paper, we want to investigate if Large language models (LLMs) can assist in accessing the learners’ concept images. We work with student data in the area of graph theory (n=182). We annotated student responses to find out whether they see graphs from a graphtheoretical perspective or rely on other concepts. In this study, we performed classifications with four different large language models to automatically classify the concept image of students. Our results show that GPT 4.0 performed best compared to the other models (F1-Score = 0.56). This demonstrates the difficulties LLMs have in classifying the concept image without fine-tuning. Future work should replicate this study applying fine-tuning or using Deep Learning models, which have shown to perform better in domain-specific settings.