Analyzing 5,665 crowd ratings on 1,133 social media comments, we find
that individuals tend to agree on the extremes of a hate rating scale
more than in the middle when evaluating the hatefulness of online
comments. The agreement is higher for less hateful comments and lowest
on moderately hateful comments. The results have implications for
researchers developing machine learning models for online hate
processing, as the extreme classes are likely to require fewer
annotations for reaching statistical stability. Our findings suggest
that the models developed in this domain should consider the
distributions of hate ratings rather than average hate scores.