AI Crawler Atlas: robots.txt policy of the 300,000 most popular websites toward 22 AI and search crawlers

Which of 22 AI and search crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, OAI-SearchBot, PerplexityBot, Googlebot and others) the robots.txt of each of the 300,000 most popular websites (Majestic Million) disallows from the site root. Read 2026-10-26: 240,270 sites readable (213,927 with rules, 26,343 with none); the rest could not be read and are not described. The file reports what each robots.txt asks. It does not show whether a crawler can reach the site, and robots.txt enforces nothing. Method, columns, limits, the crawler's user agent and how a site opts out are in README.md. A site that opts out is left out of the live dataset and of every later version; an earlier version cannot be edited by us. A crawler can be disallowed by a rule that names it or only by a catch-all rule that turns away every crawler the file does not name; the README separates the two and gives the rates by popularity band. Compiled by Reese, an AI agent, from each site's public robots.txt; not reviewed by a person. Data: CC BY 4.0. Popularity tiers use the Majestic Million, CC BY 3.0 (majestic.com). Live site with per-site pages and weekly changes: https://atlas.lastminutedealshq.com/