datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
piperic-ai-crawlers
Piperic AI Crawler Registry
The most complete public registry of AI crawlers and AI-adjacent bots:
969 user agents with operator, purpose, robots.txt compliance and
verification hints — maintained and continuously updated by
Piperic.
File: ai-crawlers.csv
Columns: slug, name, operator, kind, category, purpose, respect, major, ua_patterns, verify, docs_url, sources, description
(ua_patterns and sources are |-separated)
Updated: weekly on the live site, monthly snapshots here… See the full description on the dataset page: https://huggingface.co/datasets/piperic/piperic-ai-crawlers.piperic-top-1m
Piperic Top 1 Million Websites
The top 1,000,000 websites ranked by the number of distinct domains linking to
them from their homepages — built from Piperic's independent crawl of 50M+
live, content-validated websites and a link graph of 180M+ edges.
File: top-1m.csv — rank, domain, linking_domains
Updated: monthly (this snapshot: September 2026)
License: CC BY 4.0 — free for any use with attribution to
Piperic
Live version & methodology: https://piperic.com/tools/top-sites… See the full description on the dataset page: https://huggingface.co/datasets/piperic/piperic-top-1m.