A1 Refereed original research article in a scientific journal
How to Make Sense of Reliability? Common Language Interpretation of Reliability and the Relation of Reliability to Effect Size
Authors: Metsämuuronen, Jari; Niemensivu, Timi
Publisher: SAGE Publications
Publishing place: THOUSAND OAKS
Publication year: 2025
Journal: Applied Psychological Measurement
Journal name in source: Applied Psychological Measurement
Journal acronym: APPL PSYCH MEAS
Number of pages: 21
ISSN: 0146-6216
eISSN: 1552-3497
DOI: https://doi.org/10.1177/01466216251350159
Web address : https://doi.org/10.1177/01466216251350159
Self-archived copy’s web address: https://research.utu.fi/converis/portal/detail/Publication/498940181
Communicating the factual meaning of a particular reliability estimate is sometimes difficult. What does a specific reliability estimate of 0.80 or 0.95 mean in common language? Deflation-corrected estimates of reliability (DCER) using Somers' D or Goodman-Kruskal G as the item-score correlations are transformed into forms where specific estimates from the family of common language effect sizes are visible. This makes it possible to communicate reliability estimates using a common language and to evaluate the magnitude of a particular reliability estimate in the same way and with the same metric as we do with effect size estimates. Using a DCER, we can say that with k = 40 items, if the reliability is 0.95, in 80 out of 100 random pairs of test takers from different subpopulations on all items combined, those with a higher item response will also score higher on the test. In this case, using the thresholds familiar from effect sizes, we can say that the reliability is "very high." The transformation of the reliability estimate into a common language effect size depends on the size of the item-score association estimates and the number of items, so no closed-form equations for the transformations are given. However, relevant thresholds are provided for practical use.
Downloadable publication This is an electronic reprint of the original article. |  |
Funding information in the publication:
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study has been completed as part of the EDUCA Flagship project funded by the Research Council of Finland (#358924, #358947).
