Building an Expert Annotated Corpus of Brazilian Instagram Comments for Hate Speech and Offensive Language Detection

The understanding of an offense is subjective and people may have different opinions about the offensiveness of a comment. Also, offenses and hate speech may occur through sarcasm, which hides the real intention of the comment and makes the decision of the annotators more confusing. Therefore, provide a well-structured annotation process is crucial to a better understanding of hate speech and offensive language phenomena, as well as supply better performance for machine learning classifiers. In this paper, we describe a corpus annotation process, which was guided by a linguist, and a hate speech skilled to support the identification of hate speech and offensive language on social media. In addition, we provide the first robust corpus of this kind for the Brazilian Portuguese language. The corpus was collected from Instagram posts of political personalities and manually annotated, being composed by 7,000 documents annotated according to three different layers: a binary classification (offensive versus non-offensive language), the level of offense (highly offensive, moderately offensive, and slightly offensive messages), and the identification regarding the target of the discriminatory content (xenophobia, racism, homophobia, sexism, religious intolerance, partyism, apology to the dictatorship, antisemitism, and fat phobia). Each comment was annotated by three different annotators, and achieved high inter-annotator agreement. The new proposed annotation approach is also language and domain-independent (although it has been applied for Brazilian Portuguese).