AI Crawler Visibility and JavaScript Rendering Gaps in Small Business Websites: Dual-Capture Dataset (n=368)

This dataset accompanies the study “AI Crawler Visibility and JavaScript Rendering Gaps in Small Business Websites” by Axion Deep Digital. It measures how much answer-critical content on small business websites is available to a JavaScript-rendering engine such as Google but absent from a non-rendering AI crawler’s direct fetch, including GPTBot, ClaudeBot, and PerplexityBot. Method: Each site was captured twice in a single pass. The first capture was rendered in headless Chromium with full JavaScript execution. The second was collected as raw HTML from a single HTTP GET using a GPTBot user agent with no JavaScript execution. An element is classified as AI-invisible when it is present in the rendered DOM but absent, or below the defined threshold, in the raw fetch. Of the 405 sites attempted, 368, or 90.9%, returned analyzable dual captures and constitute the analysis set. Headline findings: • 25.0% of sites have at least one answer-critical element that is AI-invisible (95% CI: 20.8–29.7). • 14.7% hide their name, address, or phone information, collectively referred to as NAP. • Managed server-side rendering builders, Wix and Webflow, had an AI-invisible rate of 4.0%, compared with 28.3% for all other platforms combined. • The difference was statistically significant using a two-proportion z-test: z = 3.69, p < .001, representing a roughly sevenfold difference. The findings are descriptive, and the study makes no causal claims. The dataset is anonymized at the row level, with domains removed so that individual businesses are not identifiable. Columns: site_id: Anonymized identifier. vertical: Business category. platform: Detected CMS or website builder. raw_words: Word count from the raw HTTP fetch. rendered_words: Word count from the JavaScript-rendered DOM. raw_links: Link count from the raw HTTP fetch. rendered_links: Link count from the JavaScript-rendered DOM. text_ai_invisible: 1/0 flag indicating whether text was present when rendered but absent or below threshold in the raw fetch. h1_ai_invisible: 1/0 flag indicating whether the primary heading was present when rendered but absent in the raw fetch. links_ai_invisible: 1/0 flag indicating whether links were present when rendered but absent in the raw fetch. phone_ai_invisible: 1/0 flag indicating whether a phone number was present when rendered but absent in the raw fetch. email_ai_invisible: 1/0 flag indicating whether an email address was present when rendered but absent in the raw fetch. address_ai_invisible: 1/0 flag indicating whether an address was present when rendered but absent in the raw fetch. schema_ai_invisible: 1/0 flag indicating whether structured data was present when rendered but absent in the raw fetch. form_ai_invisible: 1/0 flag indicating whether a form was present when rendered but absent in the raw fetch. nap_ai_invisible: 1/0 flag indicating whether name, address, or phone information was present when rendered but absent in the raw fetch. any_critical_ai_invisible: 1 if any answer-critical element is AI-invisible. Full report and methodology: https://www.axiondeepdigital.com/research/ai-visibility-study