A4 Refereed article in a conference publication
Online hate ratings vary by extremes: A statistical analysis
Authors: Salminen J., Almerekhi H., Kamel A., Jung S., Jansen B.
Conference name: International ACM SIGIR Conference on Research and Development in Information Retrieval
Publisher: Association for Computing Machinery, Inc
Publication year: 2019
Journal: International ACM SIGIR Conference on Research and Development in Information Retrieval
Book title : CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval
Journal name in source: CHIIR 2019 - Proceedings of the 2019 Conference on Human Information Interaction and Retrieval
First page : 213
Last page: 217
Number of pages: 5
ISBN: 978-1-4503-6025-8
DOI: https://doi.org/10.1145/3295750.3298954
Analyzing 5,665 crowd ratings on 1,133 social media comments, we find
that individuals tend to agree on the extremes of a hate rating scale
more than in the middle when evaluating the hatefulness of online
comments. The agreement is higher for less hateful comments and lowest
on moderately hateful comments. The results have implications for
researchers developing machine learning models for online hate
processing, as the extreme classes are likely to require fewer
annotations for reaching statistical stability. Our findings suggest
that the models developed in this domain should consider the
distributions of hate ratings rather than average hate scores.
