CommonCrawl News Articles by Political Orientation

Dataset description & reproduction steps

The dataset includes news articles gathered from CommonCrawl for media outlets that were selected based on their political orientation. The news articles span publication dates from 2010 to 2021. For more details, please check out our Paper and GitHub repository.

The database file containing the news articles has two main tables, article_urls and article_contents. The tables have the following columns:

article_urls:

uuid: An ID that uniquely identifies this URL entry. This column is used as primary key for the table.

url: The plain text URL for the news article, as found in CommonCrawl.

outlet_name: The name of the news outlet that published the article.

article_contents:

uuid: An ID that uniquely identifies this content entry. This column is used as primary key for the table. The key is the same key used in the article_urls table to allow for cross-referencing.

date: The automatically extracted publishing date of the article. If it was not possible to automatically extract the date, this field remains empty.

content: The plain text content of the article automatically extracted from the crawled HTML document.

content_preprocessed: The articles content split by sentences.

langauge: The langauge of the article, as identified by the langdetect module (as ISO 639-1 code).