NbAiLab/aurora-robots-txt
Fetched robots.txt for creation of the Aurora dataset. Columns are url: The URL of the robots.txt file fetched content: The content of the response. Pages that responded with 404 Not Found or 410 Gone were assumed to allow all. Pages that responded with 403 Forbidden were assumed to disallow all. Pages that were tried but gave other errors contain a null value. fetched: The timestamp of the visit.
016
Conversations for this repository live on Hugging Face.
CoolFace shows imported repositories read-only. Posting into someone else’s repository from here would need an authorised integration and the account holder’s consent, so the link goes to the source instead.
Open discussions on Hugging Face