Block training, keep answers? robots.txt rules for AI bots on 24,419 domains (September 2026)

Trakis Study 1. Of the sites that keep an AI company's training bot out in robots.txt, how many also keep out the same company's bots that answer people: its search, and the pages its assistant opens when a person asks? The robots.txt files of 24,419 domains from the Tranco list of 13 October 2026 (the top 10,000; 10,000 drawn at random from ranks 100,000 to 1,000,000; every .ro domain in the list), as Common Crawl archived them in its September 2026 crawl (CC-MAIN-2026-39). No request went to the sites. 13,938 were readable, and each was judged for 14 crawlers with the open-source robots-parser library. Main results, all samples together: GPTBot is kept out of the whole site by 11.4% of readable sites, OAI-SearchBot by 3.5%. Of the 1,331 sites that block GPTBot by name, 17.1% also block OAI-SearchBot. 609 of them (45.8%) do it with Cloudflare's managed robots.txt, which names no answer bot; of the 722 that wrote the rule themselves, 31.3% also block OAI-SearchBot and 47.0% ChatGPT-User. Files: totals.csv and pairs.csv for every sample; top-10k.csv, site by site, for the 10,000 highest-ranked domains only; code.zip, the scripts that reproduce the three tables byte for byte from Common Crawl's archive. README.md gives the method, the definitions, the checks and the limits.