Robots.txt rules for AI crawlers on popular sites

For each of the top 1,000 domains of the Tranco list V354N, the answer to one request for /robots.txt on 2026-11-02, and the robots.txt verdict for 15 user agents: GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Bytespider, meta-externalagent, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Perplexity-User, Googlebot, PoliteCrawler. The data also gives the lines that decided each verdict, the content signals and the crawl delays. The crawler polite-crawler sent the requests from 2026-11-02T06:57:08.499Z to 2026-11-02T06:58:42.988Z (UTC), and decided each agent with RFC 9309. Of the 1000 domains, 509 answered with a robots.txt file and 48 with no file. 85 blocked the request at the edge, and 358 were unreachable. The files: robots-study.csv (one row for each domain), robots-study.json (the same domains, with the lines that decided), README.md (the method, the columns, the dates and the limits) and CITATION.cff. The study page tells the method, the findings and the limits: Who lets AI agents in? The robots.txt rules of the top 1,000 domains.