CoolFace
Datasetpublic

joshuarg007/ai-visibility-small-business-websites-2026

AI Crawler Visibility and JavaScript Rendering Gaps in Small Business Websites: Dual-Capture Dataset (n=368) This dataset accompanies the study by Axion Deep Digital measuring how much answer-critical content on small business websites is available to a JavaScript-rendering engine (Google) but absent from a non-rendering AI crawler's direct fetch (GPTBot, ClaudeBot, PerplexityBot). Method: Each site was captured twice in a single pass, once rendered in headless Chromium with… See the full description on the dataset page: https://huggingface.co/datasets/joshuarg007/ai-visibility-small-business-websites-2026.

sourceHugging Facecc-by-4.0updated 3mo agoView on Hugging Face
0likes24downloads
Dataset Card

AI Crawler Visibility and JavaScript Rendering Gaps in Small Business Websites: Dual-Capture Dataset (n=368)

This dataset accompanies the study by Axion Deep Digital measuring how much answer-critical content on small business websites is available to a JavaScript-rendering engine (Google) but absent from a non-rendering AI crawler's direct fetch (GPTBot, ClaudeBot, PerplexityBot).

Method: Each site was captured twice in a single pass, once rendered in headless Chromium with full JavaScript execution, and once as raw HTML from a single HTTP GET with a GPTBot user agent and no JavaScript. An element is classified AI-invisible when present in the rendered DOM but absent (or below threshold) in the raw fetch. Of 405 sites attempted, 368 (90.9%) returned analyzable dual captures.

Headline findings:

  • —25.0% of sites have at least one answer-critical element AI-invisible (95% CI 20.8-29.7)
  • —14.7% hide their name, address, or phone (NAP)
  • —Managed server-side-rendering builders (Wix, Webflow) were 4.0% invisible vs 28.3% for all other platforms (two-proportion z=3.69, p<.001, roughly sevenfold)

Descriptive; no causal claims. Anonymized at row level (domains removed).

Columns

  • —site_id: anonymized identifier
  • —vertical: business category
  • —platform: detected CMS or website builder
  • —raw_words, rendered_words: word counts from the raw fetch vs the rendered DOM
  • —raw_links, rendered_links: link counts, raw vs rendered
  • —text_ai_invisible, h1_ai_invisible, links_ai_invisible, phone_ai_invisible, email_ai_invisible, address_ai_invisible, schema_ai_invisible, form_ai_invisible, nap_ai_invisible: 1/0 flags (present when rendered, absent in the raw fetch)
  • —any_critical_ai_invisible: 1 if any answer-critical element is AI-invisible

Links and citation

  • —Full report: https://www.axiondeepdigital.com/research/ai-visibility-study
  • —Archived on Zenodo (DOI): https://doi.org/10.5281/zenodo.21312167
  • —Working paper (SSRN): https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6915818
  • —License: CC BY 4.0

Author: Joshua R. Gutierrez (ORCID 0009-0008-2595-9484), Axion Deep Digital.