The capabilities of Artificial Intelligence (AI), and specifically large language models (LLMs) have
changed the way teachers work. Using LLM or other AI-assisted tools to help review student submissions
has quickly become common practice. These AI-driven automatic assessment tools still have a lot of open
questions regarding their effectiveness, performance, and reliability. In this study, we observe the LLMs'
capabilities to assess textual answers. The data set used consisted of 31 different computer science-related
questions and 2981 answers written in English with detailed feedback and the correct answers. The LLM
we used was GPT-4o from OpenAI. At first, the performance of the LLM was tested against a single
question present in the data set producing scores for all of its answers (N=82) using multiple different
variations of settings. The best-performing approach was then used to process the full data set. With the
full set, the model got the exactly correct evaluation in 41,3% of the cases. With an accepted error margin
of ±20%, the correctness was 74.7%. When observing the fully correct answers in the set (N=1802), the
model was able to correctly identify 51.4% of them. The results can be used to guide future research
endeavors in AI-driven automatic assessment research and to guide teachers on how to improve the
performance of educational use of LLMs in different ways.