SentiAnno 2.0

Sentiment annotated gld standard corpus of Austrian histoical newspapers, covering the period of 1703-1938.Newspapers from the ANNO (Österreichischen Nationalbibliothek, 2021) and DIGITARIUM (Austrian Academy of Sciences, 2017) repositories were annotated. The corpus created in the scope of the project covers the period from 1700 to 1938, including titles such as “Wienerisches Diarium,” “Der Wanderer,” “Das Vaterland,” “Deutsches Volksblatt,” “Arbeiter Zeitung” and “Illustrierte Kronen Zeitung”. The annotations were provided by three non-expert annotators (Master’s and Bachelor’s students), drawing on established annotation methodologies (Sprugnoli et al., 2023; Schmidt et al., 2021). The annotators were employed for eight hours per week for three months each. As this exceeds the 6000 Euro project budget provided by CLARIAH-AT, the additional staff costs were covered by the Department of Digital Humanities. The project PI prepared the annotations and the annotation guidelines. According to best practices (Sprugnoli et al., 2023; Schmidt et al., 2021), frequent project meetings took place to ensure guideline understanding. The annotations were collected using Doccanno (Hiroki et al., 2018) and evaluated for inter-annotator agreement to ensure quality. Over 2500 instances have been annotated, covering topics like “migration”, “labour”, “Jews,” “Czech language”, “nationalism”, “antisemitism”, “workers rights” and others. Results show a high inter-annotator percentage agreement (overall 93% for two out of three annotators) despite the arduous annotation process, historical context and complex language. Further, the Fleiss’ kappa is 0.40, demonstrating fair annotator agreement. This is consistent with other annotation studies of DH data (Schmidt et al., 2021). It is important to note that Fleiss’ kappa is extremely sensitive to uneven distribution of labels, which is a characteristic of this corpus (see Image 1). The results show that migration and minorities were often portrayed in a negative light, while neutral reporting was also frequent. The annotator experience was evaluated using a custom questionnaire and two standardized usability tests - NASA-TLX (Hart & Staveland, 1988) and UEQ-S (Hinderks et al., 2018). The results of this questionnaire demonstrate that the task of annotation of historical texts is difficult and requires taking frequent breaks. The NASA-TLX score showed an average of a moderately demanding perceived workload (55) and the average UEQ-S scores indicate a positive user experience overall (PQ: 0.92, HQ: 0.67), with high agreement on pragmatic quality and more variance in hedonic responses. The creation of this corpus was partially funded by CLARIAH-AT, through a small project "Sentiment-annotated corpus of Austrian historical newspapers".