Comparing the utility of different classification schemes for emotive language analysis

We investigated the utility of different classification schemes for emotive language analysis. We compared six schemes: (1) Ekman's six basic emotions, (2) Plutchik's wheel of emotion, (3) Watson and Tellegen's Circumplex theory of affect, (4) the Emotion Annotation Representation Language (EARL), (5) WordNet-Affect, and (6) free text classification scheme. To measure their utility, we investigated their ease of use by human annotators as well as the performance of supervised machine learning when these schemes were used to annotate the training data. We assembled a corpus of 500 emotionally charged tweets. To ensure that the text contained emotion, we collected tweets based on their inclusion of emoticons, hashtags including emotion terms, idioms, and tweets with an automatically generated sentiment. We also include emotionally neutral or ambiguous tweets while correcting for bias towards certain emotions based on the choice of idioms, emoticons and hashtags. The corpus was annotated manually using an online crowdsourcing platform (CrowdFlower) by five independent annotators per text document, per classification scheme. The data provided here consists of the annotator id (their IP address), the annotation given, and the text document from the corpus, per classification scheme.