kareenamehta/ccnews

---
language:
- multilingual
- af
- am
- ar
- as
- az
- be
- bg
- bn
- br
- bs
- ca
- cs
- cy
- da
- de
- el
- en
- eo
- es
- et
- eu
- fa
- fi
- fr
- fy
- ga
- gd
- gl
- gu
- ha
- he
- hi
- hr
- hu
- hy
- id
- is
- it
- ja
- jv
- ka
- kk
- km
- kn
- ko
- ku
- ky
- la
- lo
- lt
- lv
- mg
- mk
- ml
- mn
- mr
- ms
- my
- ne
- nl
- 'no'
- om
- or
- pa
- pl
- ps
- pt
- ro
- ru
- sa
- sd
- si
- sk
- sl
- so
- sq
- sr
- su
- sv
- sw
- ta
- te
- th
- tl
- tr
- ug
- uk
- ur
- uz
- vi
- xh
- yi
- zh
pretty_name: All of Common Crawl News, 100+ languages, preprocessed and cleaned
task_categories:
- text-classification
- question-answering
- text-generation
- text2text-generation
size_categories:
- 100M<n<1B
tags:
- news
configs:
- config_name: "default"
data_files: "*.parquet"
- config_name: "2016"
data_files: "2016*"
- config_name: "2017"
data_files: "2017*"
- config_name: "2018"
data_files: "2018*"
- config_name: "2019"
data_files: "2019*"
- config_name: "2020"
data_files: "2020*"
- config_name: "2021"
data_files: "2021*"
- config_name: "2022"
data_files: "2022*"
- config_name: "2023"
data_files: "2023*"
- config_name: "2024"
data_files: "2024*"

---

This dataset is the result of processing all WARC files in the [CCNews Corpus](https://commoncrawl.org/blog/news-dataset-available), from the beginning (2016) to June of 2024.
The data has been cleaned and deduplicated, and language of articles have been detected and added. The process is similar to what HuggingFace's [DataTrove](https://github.com/huggingface/datatrove) does.

Overall, it contains about 600 million news articles in more than 100 languages from all around the globe.

For license information, please refer to [CommonCrawl's Terms of Use](https://commoncrawl.org/terms-of-use).

Sample Python code to explore this dataset:

```python
from datasets import load_dataset
from tqdm import tqdm

# Load the news articles **crawled** in the year 2016 (but not necessarily published in 2016), in streaming mode
dataset = load_dataset("stanford-oval/ccnews", name="2016", streaming=True) # `name` can be one of 2016, 2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024

# Print information about the dataset
print(dataset)

# Iterate over a few examples
print("\nFirst few examples:")
for i, example in enumerate(dataset["train"].take(5)):
print(f"Example {i + 1}:")
print(example)
print()

# Count the number of articles (in 2016)
row_count = 0
for _ in tqdm(dataset["train"], desc="Counting rows", unit=" rows", unit_scale=True, unit_divisor=1000):
row_count += 1

# Print the number of rows
print(f"\nTotal number of articles: {row_count}")

# Extract all Arabic (ar) articles
for row in tqdm(dataset["train"], desc="Extracting articles", unit=" rows", unit_scale=True, unit_divisor=1000):
if row["language"] == "ar":
print(row)
```