Datasetpublic

osamamumtaz01/ai-crawler-user-agents

AI Crawler User Agents Machine-readable list of all 28 known AI crawler and agent user-agent strings — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Applebot-Extended, and more — with each bot's operator, purpose, observed robots.txt compliance, and documented crawl-delay support. Fields Field Description userAgent Token to match in robots.txt / server logs (e.g. GPTBot) operator Company running the crawler purpose… See the full description on the dataset page: https://huggingface.co/datasets/osamamumtaz01/ai-crawler-user-agents.

sourceHugging Facecc-by-4.0updated 10d agoView on Hugging Face

0likes40downloads

Dataset Card

AI Crawler User Agents

Machine-readable list of all 28 known AI crawler and agent user-agent strings — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, Applebot-Extended, and more — with each bot's operator, purpose, observed robots.txt compliance, and documented crawl-delay support.

Fields

Field	Description
`userAgent`	Token to match in robots.txt / server logs (e.g. `GPTBot`)
`operator`	Company running the crawler
`purpose`	`training`, `search`, or `both`
`description`	One-line summary
`robotsCompliance`	`yes` / `partial` / `no` / `unknown` — observed behavior, not just vendor claims
`robotsNote`	Sourced explanation of the rating
`docsUrl`	Official documentation, where published
`ipRangeUrl`	Operator-published IP-range JSON, where one exists — verify the crawler by IP instead of trusting the user-agent header
`crawlDelay`	`yes` / `no`, or absent — see the note below
`crawlDelayNote`	The operator's own wording, where they state one

A note on `crawlDelay`

Only 4 of the 28 operators state any position on the crawl-delay directive, and two of them support it:

Crawler	crawl-delay	Source
YouBot	`yes`	You.com documents that it honours crawl-delay directives
ImagesiftBot	`yes`	Reads the value as the minimum seconds between consecutive requests, and documents the interval arithmetic
Amazonbot	`no`	"They do not support the crawl-delay directive"
Applebot	`no`	"Applebot does not follow crawl-delay"

The other 24 publish nothing either way, and the field is absent for them rather than set to no. In practice you should treat undocumented as unsupported — crawl-delay was never in the robots.txt specification and Google ignores it — but that is an assumption, not a finding, and the dataset keeps the two distinct so you can tell which one you are looking at.

Use cases

—Generate robots.txt rules programmatically (block training bots, keep search bots)
—Detect AI crawler traffic in server logs or middleware
—Research on AI crawler behavior and robots.txt compliance

Source & freshness

Reviewed monthly. Always-current endpoint (CORS enabled): <https://geoprompttracker.com/data/ai-crawlers.json>

Human-readable directory with per-bot robots.txt rules: <https://geoprompttracker.com/bots>

GitHub mirror: <https://github.com/osamamumtaz01/ai-crawler-user-agents>

Attribution