Datasetpublic

NbAiLab/aurora-robots-txt

Fetched robots.txt for creation of the Aurora dataset. Columns are url: The URL of the robots.txt file fetched content: The content of the response. Pages that responded with 404 Not Found or 410 Gone were assumed to allow all. Pages that responded with 403 Forbidden were assumed to disallow all. Pages that were tried but gave other errors contain a null value. fetched: The timestamp of the visit.

sourceHugging Faceupdated 6mo agoView on Hugging Face

0likes21downloads

Dataset Card

Fetched robots.txt for creation of the Aurora dataset.

Columns are

—url: The URL of the robots.txt file fetched
—content: The content of the response. Pages that responded with 404 Not Found or 410 Gone were assumed to allow all. Pages that responded with 403 Forbidden were assumed to disallow all. Pages that were tried but gave other errors contain a null value.
—fetched: The timestamp of the visit.