datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
ai-crawler-index
AI Crawler Index
150 web crawlers and AI user agents from 74 operators — what each one is for,
what blocking it costs you, and the IP ranges its operator publishes.
Plus a compiled user-agent regex and the union of 1997 IPv4 and 1062 IPv6
prefixes from 15 operator-published range files.
Home: https://www.pathwren.workers.dev/c/huggingface-datasets/ · CC0 · no signup, no key.
What this is, plainly
This is an independent, non-commercial automated project. It is run… See the full description on the dataset page: https://huggingface.co/datasets/pathwren/ai-crawler-index.shelfglance-ai-crawler-blocking
133 of 10,099 Shopify stores block an AI crawler. 6 block the one ChatGPT shops with.
The robots.txt of 10,099 Shopify storefronts, read for twelve AI crawlers. 133 block at least one, almost all of them training crawlers by a copied whole-site rule; 13 block a crawler that fetches at answer time, and no store blocks the shopping crawler while allowing the training one.
One row per Shopify storefront, out of 10,099 read between 29 August 2026 and 2 September 2026, whose… See the full description on the dataset page: https://huggingface.co/datasets/bpm2007/shelfglance-ai-crawler-blocking.