A4 Refereed article in a conference publication
ChatGPT as tutor? A case study on competitive programming
Authors: Rytilahti, Juuso; Lokkila, Erno
Editors: Carmo, Mafalda
Conference name: International Conference on Education and New Developments
Publication year: 2024
Journal: Education and New Developments
Book title : Education and New Developments 2024: Volume 2
First page : 91
Last page: 95
ISBN: 978-989-35728-0-1
ISSN: 2184-044X
eISSN: 2184-1489
DOI: https://doi.org/10.36315/2024v2end019
Web address : https://doi.org/10.36315/2024v2end019
In this paper, we present a case study on how students utilize ChatGPT as a tutor for a short competitive programming course. The students were divided into two groups, one led by a teacher, and the other tutored by ChatGPT. The course was an intensive five-day course and both groups studied concurrently. Students could freely choose which group they participated in. The ChatGPT group was provided a guide on the basics of prompting, including approaches on how to generate the study material. Both groups were allowed to use any learning material, and only the teacher-led group excluded the use of generative AI tools. Research questions: In this study, we focus on the following questions: (1) How did students approach using ChatGPT as a tutor?; (2) Are there significant differences between students led by a teacher or by ChatGPT?; (3) How did students in both groups experience the teaching and did it correlate to the achieved learning results (exam)? Methodology: The data consists of survey data, and final exam given to students (N=11). We also collected the discussion history of the ChatGPT group. The discussion history was divided into prompt-message pairs (N=340) and analyzed. The data was analyzed using mixed methods. The discussion history was analyzed using grounded theory. Statistical methods were used to find any correlation between initial skill level and learning as well as the tag distribution of the ChatGPT discussions. Results: Differences were found between the two groups. Those with a higher initial skill level seemed to favor the ChatGPT group, whereas the less experienced chose the in-person teaching. Analysis of the ChatGPT discussion history showed mostly similar usage patterns across students. We present the distribution of tags used by the ChatGPT group. Additionally, we give insight on how to approach similar research settings in the future. Impact: All around the world, students are already utilizing ChatGPT as a substitute for a teacher or a tutor. Our pilot study provides insight into how students approach utilizing ChatGPT as a tutor in a programming teaching setting. These preliminary results can be used to guide future research settings.