commoncrawl-urls

# **Common Crawl URL Seed** > Every URL in a Common Crawl snapshot, as partitioned Parquet, ready to reseed a crawl ## What is it? This dataset is the full URL list of a [Common Crawl](https://commoncrawl.org) snapshot, pulled straight from the crawl's columnar URL index and written as partitioned Parquet. Common Crawl is a non-profit that crawls the web and freely publishes its archives. Each snapshot ships a columnar index whose `url` column names every page the crawl captured. This dataset is that column, deduplicated and sharded, so a downstream crawler can reseed from a known frontier without pulling the 150 GB+ index again. It currently holds crawl **CC-MAIN-2026-25** (warc subset) with **2,098,491,742 URLs across 256 shards**. The dataset is released under the **Open Data Commons Attribution License (ODC-By) v1.0**, the same license Common Crawl uses. ## What is being released? The URLs are sharded by host. Each shard is the set of URLs whose host hashes into one contiguous slice of the 64-bit hostkey space, so every URL of a host lands in exactly one shard and the shards tile the whole space with no gap and no overlap. That is the same partitioning [meguri](https://github.com/tamnd/meguri) assigns when it ingests a seed, so a shard maps one to one onto a crawl-frontier partition. The shards live under a crawl-partitioned directory: ``` data/ crawl=CC-MAIN-2026-25/ shard-00000.parquet shard-00001.parquet shard-00002.parquet ... manifest.json ``` `manifest.json` is the shard map: it records each shard's hostkey range and row count, so a puller can rebuild the exact same frontier partitions the crawl used. ## Schema | Column | Type | Description | |---|---|---| | `url` | string | A URL captured by the crawl | | `host` | string | Host parsed from the URL, the key the shard is partitioned on | ## How to load it ### Using `datasets` ```python from datasets import load_dataset # stream every URL in the crawl ds = load_dataset("open-index/commoncrawl-urls", name="CC-MAIN-2026-25", split="train", streaming=True) for row in ds: print(row["url"]) # load one shard into memory ds = load_dataset( "open-index/commoncrawl-urls", data_files="data/crawl=CC-MAIN-2026-25/shard-00000.parquet", split="train", ) ``` ### Using DuckDB ```sql SELECT url FROM read_parquet('hf://datasets/open-index/commoncrawl-urls/data/crawl=CC-MAIN-2026-25/**/*.parquet') WHERE host = 'en.wikipedia.org'; ``` ### Reseed a crawl frontier Download the shards and hand them to meguri to rebuild the crawl frontier without touching Common Crawl again: ```bash huggingface-cli download open-index/commoncrawl-urls --repo-type dataset \ --include "data/crawl=CC-MAIN-2026-25/**" --local-dir ./cc-urls # the parquet url column is the seed; feed it straight into a meguri store meguri shard build --urls ./cc-urls/data/crawl=CC-MAIN-2026-25 --out ./frontier ``` ## Provenance The URLs come from the Common Crawl columnar URL index (`cc-index/table/cc-main/warc`), subset `warc`, for crawl `CC-MAIN-2026-25`. Only the `url` column of the index is read; nothing about page content is included here. The pull, the host sharding, and this dataset are produced by [ccrawl-cli](https://github.com/tamnd/ccrawl-cli). ## Sizes Measured across 256 shards of CC-MAIN-2026-25. | Stage | Size | |---|---| | URLs | 2,098,491,742 | | URL text (uncompressed) | 149.4 GB | | Parquet (Zstd) | 31.0 GB | ## Licensing Released under the **Open Data Commons Attribution License (ODC-By) v1.0**. Use is also subject to [Common Crawl's Terms of Use](https://commoncrawl.org/terms-of-use). The URLs point at content that remains subject to its publishers' rights.