NewsUnravel Dataset

<strong>About the Dataset</strong><br> Media bias is a multifaceted problem, leading to one-sided views and impacting decision-making. A way to address bias in news articles is to automatically detect and indicate it through machine-learning methods. However, such detection is limited due to the difficulty of obtaining reliable training data. To facilitate the data-gathering process, we introduce NewsUnravel, a news-reading web application leveraging an initially tested feedback mechanism to collect reader feedback on machine-generated bias highlights within news articles. Our approach augments dataset quality by significantly increasing inter-annotator agreement by 26.31% and improving classifier performance by 2.49%. As the first human-in-the-loop application for media bias, NewsUnravel shows that a user-centric approach to media bias data collection can return reliable data while being scalable and evaluated as easy to use. NewsUnravel demonstrates that feedback mechanisms are a promising strategy to reduce data collection expenses, fluidly adapt to changes in language, and enhance evaluators' diversity.<br> <br> <strong>Description of the data files</strong><br> This repository contains the datasets for the anonymous NewsUnravel submission. The tables contain following data:<br> <br> <strong>Feedback.csv</strong>: holds the participantID of a single feedback with the sentence ID (contentId), the bias rating, and provided reasons<br> <strong>Content.csv</strong>: holds the participant ID of a rating with the sentence ID (contentId) of a rated sentences and the bias rating, and reason, if given<br> <strong>Article.csv</strong>: holds the article ID, title, source, article meta data, article topic, and bias amount in %<br> <strong>Participant.csv</strong>: holds the participant IDs and data processing consent