Datasetpublic

phl-ldm/tranco_top_million

Top 1M Tranco Crawl Dataset A web crawl dataset covering the top 1 million domains from the Tranco list with ID "PYGVJ" (generated July 29, 2026), collected on July 30-31, 2026 using the distcrawl crawler. The crawling process took around 18.5 hours to complete, using ~100 parallel nodes deployed worldwide. The cloud costs for data collection totaled around €20. Success Rate of the Crawl Percentage of the 1 million target domains that were successfully crawled… See the full description on the dataset page: https://huggingface.co/datasets/phl-ldm/tranco_top_million.

sourceHugging Facecc-by-4.0updated 2mo agoView on Hugging Face

0likes461downloads

Dataset Card

Top 1M Tranco Crawl Dataset

A web crawl dataset covering the top 1 million domains from the Tranco list with ID "PYGVJ" (generated July 29, 2026), collected on July 30-31, 2026 using the distcrawl crawler. The crawling process took around 18.5 hours to complete, using ~100 parallel nodes deployed worldwide. The cloud costs for data collection totaled around €20.

Success Rate of the Crawl

Percentage of the 1 million target domains that were successfully crawled (present in site_metadata.parquet):

Successful sites	Success rate
Successful sites	762,110
Success rate	76.21%

Crawl Configuration

Configuration choice	Value
Browser Engine	Chromium
Auto Accept Cookies	Enabled
Execution Mode	Headless
Dwell Time [s]	10
Max Crawl Depth	0
Scroll Sequence [px]	100, 500, -500, -100

—Browser Version: Google Chrome for Testing 148.0.7778.96 (Headless Shell)
—Uses User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/148.0.7778.96 Safari/537.36

Worker Nodes Metadata

Basic worker metadata (country, connection type based on IP address) are included, so filtering per worker country is possible.

File Structure

Each crawl configuration contains a metadata.json file with the exact crawl parameters, and an enriched/ directory with five Parquet tables:

`labeled_requests.parquet`

Every outgoing HTTP request observed during the crawl, labeled against several filter lists available on easylist.to (based on the latest available filter lists on July 31, 2026).

Column	Type	Description
`experiment_id`	string	Crawl configuration identifier
`request_id`	string	Request identifier (unique within a `crawl_session_id`)
`worker_id`	string	Worker that performed the crawl
`crawl_session_id`	string	Session ID for a single domain visit
`timestamp`	string	Request timestamp
`crawled_url`	string	Seed URL being crawled
`current_page_url`	string	Page URL that triggered the request
`url`	string	Requested URL
`frame_url`	string	Frame URL that initiated the request
`resource_type`	string	Resource type (script, image, fetch, document, xhr, etc.)
`method`	string	HTTP method
`headers`	string	Request headers (JSON)
`crawl_depth`	int	Navigation depth from the landing page
`blocked_by`	string	Filter lists that match this request (JSON array, e.g. `["easylist", "easyprivacy"]`; empty `[]` = not blocked)

`responses.parquet`

HTTP responses received during the crawl.

Column	Type	Description
`experiment_id`	string	Crawl configuration identifier
`request_id`	string	Matching request identifier
`crawl_session_id`	string	Session ID for a single domain visit
`timestamp`	string	Response timestamp
`crawled_url`	string	Seed URL being crawled
`url`	string	Response URL
`status`	int	HTTP status code
`headers`	string	Response headers (JSON)
`crawl_depth`	int	Navigation depth from the landing page
`cookies`	string	Cookies set by the response

`cookie_warning_consents.parquet`

Pages where a cookie consent dialog was detected and clicked.

Column	Type	Description
`experiment_id`	string	Crawl configuration identifier
`crawl_session_id`	string	Session ID for a single domain visit
`crawled_url`	string	Seed URL being crawled
`url`	string	Page URL where consent was given
`timestamp`	int	Consent timestamp
`crawl_depth`	int	Navigation depth from the landing page

`site_metadata.parquet`

Metadata extracted from each visited site.

Column	Type	Description
`crawl_session_id`	string	Session ID for a single domain visit
`description`	string	Site meta description
`timestamp`	string	Collection timestamp

`worker_metadata.parquet`

Information about the distributed crawl workers.

Column	Type	Description
`worker_id`	string	Worker identifier
`country_code`	string	Worker geographic location
`is_residential`	bool	Whether the worker uses a residential IP
`timestamp`	double	Registration timestamp