Capturing the Viewpoint Dynamics in the News Domain

In this data repository we deposited the gold standard for our study titled "<i>Capturing the Viewpoint Dynamics in the News Domain</i>"and having abstract:"Despite the seismic changes brought about by the web and social media, mainstream news sources still play a crucial role in democratic societies. In particular, a healthy democracy requires a balanced and diverse media landscape, providing an arena in which the various <i>topics</i> and <i>viewpoints</i> relevant to the political discourse of the day are presented and discussed. Unfortunately, there is currently little effective computational support available to the various classes of users, who are interested in monitoring the topic and viewpoint dynamics in the news – e.g., for regulatory or research purposes. As a result, current analyses by researchers and practitioners tend to be small scale and, by and large, rely on manual investigations of topic and viewpoint coverage. To address this issue, we have developed a hybrid human-machine approach, which uses a Large Language Model (LLM) first to help analysts to identify the range of viewpoints<i> </i>relevant to the debate around a given topic, and then to classify the claims expressed in the news corpus of interest with respect to the identified viewpoints. We tested a variety of LLMs on a benchmark corpus of news items drawn from British media sources and our results indicate that that the larger models can provide effective support for this classification task, even when run in a zero-shot learning modality."<br><b>Details of the Gold Standard</b>To create a reliable gold standard for evaluating various large language models, we selected a random sample of 402 claims. Five human annotators were tasked with classifying each claim according to nine viewpoints relevant to the UK immigration debate:Immigration as a management issueImmigrants as victims / Humanitarian emphasisImmigrants as potential criminals or threat / National security emphasisEnhancing / maintaining immigration pathwaysRestricting immigration pathwaysEconomic benefits of immigrationEconomic cost of immigrationIntegration policies/Multiculturalism as a positive forceAnti-integration policies/Cultural identity preservation.Each claim was rated by three annotators, and a majority vote determined the final classification, forming the "gold standard".The first gold standard (<b>gold-standard-full-9-dim.csv</b>) was created using a simple majority rule applied to all 402 annotated utterances. This required at least two annotators to agree on the classification of a claim. Our analysis showed that viewpoint 8 (Integration policies/Multiculturalism as a positive force) was particularly problematic, with its opposite, viewpoint 9 (Anti-integration policies/Cultural identity preservation) exhibiting also a low agreement score. Therefore, we also produced a gold standard with only seven viewpoints, removing viewpoints 8 and 9: <b>gold-standard-full-7-dim.csv</b>.However, only moderate agreement was achieved among annotators due to the inherent ambiguity and difficulty in interpreting political statements. Very often statements by politicians are ambiguous and difficult to interpret, and therefore, despite putting significant effort in calibrating and harmonizing the scores from different annotators, only a moderate level of agreement could be achieved. This situation causes a problem because it is highly likely that the contested nature of the domain will impact negatively on the overall performance of a LLM. For this reason, we also produced a restricted version of the gold standard (<b>gold-standard-restricted-7-dim.csv</b>). Only utterances with unanimous agreement on a negative flag or at least two agreements on a positive flag were retained. This resulted in a refined dataset of 219 utterances, ensuring a high degree of robustness and substantial agreement among annotators.<b>Files details</b>Each file is structured in the same way. The first two columns contain the utterance and the corresponding author. Subsequent columns, nine in <b>gold-standard-full-9-dim</b> and seven in the rest, denote the utterance's categorisation into predefined viewpoints, with 1 for "Yes" and 0 for "No". Each row reports an utterance. <b>gold-standard-full</b> file contain all 402 annotated utterances, whereas <b>gold-standard-restricted</b> contains a subset of 219.