Evaluating firearm examiner testimony using large language models: a comparison of standard and knowledge-enhanced AI systems




Pompedda; Francesco; Santtila, Pekka; Di Maso, Eleonora; Nyman; Thomas J.; Yongjie, Sun; Zappala, Angelo

PublisherTaylor & Francis online

2025

Journal of Psychology and AI

2503343

1

1

2997-4100

DOIhttps://doi.org/10.1080/29974100.2025.2503343

https://doi.org/10.1080/29974100.2025.2503343

https://research.utu.fi/converis/portal/detail/Publication/492228205



This study evaluated the decision-making of Large Language Models (LLMs) in interpreting firearm examiner testimony by comparing a standard LLM to one enhanced with forensic science knowledge. The present study is a replication study. We assessed whether LLMs mirrored human decision patterns and if specialised knowledge led to more critical evaluations of forensic claims. We employed a 2 × 2 × 7 between-subjects design with three independent variables: LLM configuration (standard vs. knowledge-enhanced), cross-examination presence (yes vs. no), and conclusion language (seven variations). Each model condition performed 200 repetitions per scenario. This yielded a total of 5,600 measures of binary verdicts, guilt probability ratings, and credibility assessments. LLMs showed low conviction rates (9.4%) across conditions, with logical variations as a function of the way in which the firearm expert’s conclusion was formulated. Cross-examination produced lower guilt assessments and scientific credibility ratings. Importantly, knowledge-enhanced LLMs demonstrated significantly more conservative evaluations of firearm evidence across all match conditions compared to standard LLMs. LLMs, particularly when enhanced with domain-specific knowledge, showed advantages in evaluating complex scientific evidence compared to human jurors in Garrett et al. (2020), suggesting potential applications for AI systems in supporting legal decision-making.


The work was supported by the New York University Shanghai [10109_Academic Research]; Spring Crocus s.r.l., managing body of CBT. ACADEMY.


Last updated on 2025-06-06 at 09:52