News Ninja Dataset

This repository contains the data for the News Ninja Game submission.<br> <br> <strong>About</strong><br> Recent research shows that visualizing linguistic media bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher quality than expert and crowdsourced datasets. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, enhance evaluators' diversity, and promote long-term bias mitigation.<br> <br> <strong>Content<br> demographics.csv: </strong>Contains demographic information of News Ninja players.<br> <strong>news_ninja_dataset.csv</strong>: Contains the sentences that News Ninja players labeled with their corresponding bias label, biased words, id, and annotation count.