AI SEO Crawlability & Keyword Dataset

The dataset is designed to evaluate website crawlability and accessibility for AI bots and search engine crawlers. Each record represents a domain along with its crawlability score and permissions for various AI-driven bots. Dataset Fields ExplanationDomainThe website domain being analyzed for crawlability and bot access.Crawlability Score (%)A percentage-based score indicating how accessible the website is for crawlers and AI bots. Higher scores represent better accessibility and SEO readiness.AI InsightAI-generated analysis highlighting crawl issues, restrictions, or optimization opportunities for the domain.Robots.txt LinkDirect link to the website’s robots.txt file used to control crawler access. AI Bot Accessibility StatusThese fields indicate whether specific AI bots are Allowed / Blocked / Restricted based on robots.txt rules:GPTBot StatusAccess status for OpenAI’s web crawler.ClaudeBot StatusAccess status for Anthropic’s Claude AI crawler.PerplexityBot StatusIndicates whether Perplexity AI can crawl the site.Google-Extended StatusControls access for Google’s AI training crawler.CCBot StatusAccess for Common Crawl bot (used in many AI datasets).anthropic-ai StatusAdditional Anthropic-related crawler access.OAI-SearchBot StatusStatus of OpenAI’s search-focused crawler.⚙️ Purpose of the DatasetThis dataset helps to:Analyze AI bot crawl permissions across websitesIdentify restrictions in robots.txt affecting AI visibilityMeasure overall crawlability score for SEO and AI indexingOptimize websites for AI search engines and LLM visibilityDetect opportunities to improve AI-driven traffic and discoverability Use CasesAI SEO optimizationLLM (ChatGPT, Claude, Perplexity) visibility analysisrobots.txt audit and fixingCompetitive analysis of AI crawl accessBuilding AI-ready websitesWebsite: https://dofollo.ai/