bpm2007/shelfglance-ai-crawler-blocking

133 of 10,099 Shopify stores block an AI crawler. 6 block the one ChatGPT shops with.

The robots.txt of 10,099 Shopify storefronts, read for twelve AI crawlers. 133 block at least one, almost all of them training crawlers by a copied whole-site rule; 13 block a crawler that fetches at answer time, and no store blocks the shopping crawler while allowing the training one.

One row per Shopify storefront, out of 10,099 read between 29 August 2026 and 2 September 2026, whose robots.txt names at least one of twelve AI crawlers, or which carries a noai meta tag. Each crawler column says whether that user agent is blocked for /products/ paths. The 10,099 stores that block nothing are the denominator and are not rows.

Method: robots.txt was fetched once per store and evaluated for each user agent against the /products/ path, using the standard longest-match rule. A store that blocks by other means (WAF, IP, bot management) is invisible to this reading, so every count is a lower bound.

Columns: host, grade, scannedat, OAI-SearchBot, GPTBot, ChatGPT-User, PerplexityBot, Perplexity-User, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, Bytespider, CCBot, noaimeta, scope.

The study that reads these rows, with every figure recomputed from them: https://shelfglance.com/research/ai-crawler-blocking The rows as served by the site: https://shelfglance.com/research/ai-crawler-blocking.csv

Licence: Creative Commons Attribution 4.0 (https://creativecommons.org/licenses/by/4.0/). Read on 2 September 2026; version 2026-09-02.

Rows: 139. File: shelfglance-ai-crawler-blocking.csv.