Dead-Web Index — reachability census of the top 10 million domains
收藏资源简介:
A reachability census of the DomCop top 10 million domains (June 2026). Every domain is probed and labelled alive / redirect / blocked / dead twice — once by a polite plain-HTTP bot, and once by a reachability client with a real Chrome TLS/JA3 fingerprint. The dataset behind the Crawlora Dead-Web Index explorer. Headline: 14.1% truly dead (no DNS / refused / nothing listening); 8.9% blocked but alive (a live server returning 403/429/anti-bot); 76.6% alive, 0.3% redirect. A real Chrome TLS fingerprint reaches ~72,000 domains the polite bot cannot. Files: 10 rank-partitioned zstd-JSONL chunks per arm (polite: 9,992,781 rows; reachability: 9,997,315 rows), a 1,000-row uncompressed sample, a checksummed manifest, and run-level aggregates. One JSON record per line (domain, tld, rank, mode, outcome, reason, statuses, final_url, scheme, hops, parked, run_id, probed_at). License: CC BY 4.0. Source: github.com/Crawlora-org/dead-web-index-data; explorer: crawlora.net/dead-web-index; methodology: crawlora.net/blog/how-much-of-the-web-is-dead-2026.



