Comparison of two annotation schemes to derive offensiveness scores in HurtLex.

The recent proliferation of social media has been accompanied by a similar widespread increase in the frequency of hostile online messages and inflammatory speeches. Social media are nowadays seen as a suitable arena for ill-tempered debates to be conducted, with the frequent use of insulting terms and other offensive language. Lexical resources containing specific negative words have been widely employed to detect uncivil communication. This work describes the development, implementation and comparison of two annotation schemes to derive offensiveness scores in a lexicon of hate words, namely HurtLex. Since determining a given expression’s offensiveness is a highly subjective matter, we decided to use two different methods to derive offensiveness scores. The first method consists in a rating scale annotation such that each word has to be assessed on a 5-point Likert scale and to derive a final score we explicitly model the item response probability trough a unidimensional Graded response Model (GrM). The second one is the Best-worst scaling method. we analyse whether these two different methods lead to correlated scores. we propose a qualitativequantitative analysis of the terms that have opposite scores. The two scoring systems are evaluated on two annotated corpora. The introduction of an offensiveness level score in HurtLex could be helpful as part of a lexicon-based approach to track offensive and hateful content