An Experimental Annotation Task Investigating Annotator Agreement Within a Misogynistic Dictionary and Corpus

This work describes the development of two distinct experimental annotation tasks aimed at examining the agreement among annotators in two different contexts of misogyny: a misogynistic dictionary and a corpus consisting of misogynistic comments. In these two tasks, each annotator is required to categorize every term of a dictionary (or a comment) as offensive or misogynistic. Additionally, if a term/comment is deemed misogynistic, annotators should further specify its subcategory, including sexual objectification, dominance, body-shaming, derogatory language, intimidation, benevolent sexism, and neo-sexism. We compare different measures to assess the level of agreement among annotators.