AsciiMAster/pl-web-graph-2026-09-14

Polish Web Domain Observations 2026-09-14

A curated snapshot of a .pl-focused domain crawler: DNS observations, host availability metadata, discovered URL references, and the crawl frontier. No HTML, page text, or website classifications are included. Observations accumulated over months, so September 14 dates the export itself while each row carries its own observation time. Coverage is whatever one crawler reached, and liveness holds as of the recorded timestamp.

[image]

Rendered from this snapshot's link graph with the project's `graph_viz.py`. The picture is heavily filtered. It draws the largest connected component alone, capping each registrable domain at 200 hostnames so that no single operator takes over the frame — one link farm by itself mints over 179,000 hostnames. `.gov.pl` is exempt from that cap, since its host count reflects many institutions sharing a zone. The `graph_edges` subset holds far more than what is drawn. Every node is an integer id, so no hostname reaches the image.

License: [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/). The publisher dedicates this compilation to the public domain, waiving copyright and, to the extent held, the EU/Polish sui generis database right arising from the effort of collecting it. No attribution is required, though citation is appreciated.

CC0 reaches only the rights the publisher holds in this compilation. It grants nothing over the third-party websites these records describe, their content, or their trademarks, and as a copyright instrument it leaves data-protection obligations untouched. Records such as ip, and hostnames that embed personal names, may constitute personal data under the GDPR; the redacted-* values are pseudonyms, which the GDPR treats as personal data rather than anonymous. Reusers remain responsible for their own lawful basis for processing.

Choose a subset

• —hosts is the default: one row per recorded site, with DNS, zone, page-frontier and graph summaries already joined.
• —graph_edges is a directed host-to-host graph, grouped by reference kind. Choose kind == "link" for the recorded hyperlink graph. Record counts measure how many link rows the crawler stored for that pair, which is crawl evidence rather than popularity or traffic.
• —zone_summary provides independently aggregated DNS, host and frontier statistics. A null zone, where present, collects frontier hosts whose zone matched nothing.
• —source_* retains all rows and original columns from each of the seven SQLite tables, with explicit types and sanitization. source_domains holds legacy zone checks and source_restored_sources holds recovery bookkeeping; both are internal bookkeeping, and sites remains the host inventory.

Every subset uses one train split holding the complete snapshot. That name follows the loading convention; the data carries no labels and no train/test partition.

Load and analyze

from datasets import load_dataset

hosts = load_dataset("AsciiMAster/pl-web-graph-2026-09-14", name="hosts", split="train", streaming=True)
print(next(iter(hosts)))

import duckdb

# Query Parquet directly without loading all rows into Python memory.
print(duckdb.sql("""
    SELECT zone, count(*) AS responding_hosts
    FROM read_parquet('data/hosts/*.parquet')
    WHERE responded AND valid
    GROUP BY zone ORDER BY responding_hosts DESC LIMIT 20
""").df())

import pandas as pd
zones = pd.read_parquet("data/zone_summary")
print(zones.nlargest(10, "responding_hosts")[["zone", "responding_hosts"]])

Run python examples/analyze.py from this dataset directory for all three examples. Large subsets support streaming and column projection; keep the full link table out of pandas.

How the subsets join

No column name is shared by every subset. They join on the hostname, which carries a different name in each:

Start from hosts and join outward. The coarser axis is the zone: hosts.zone matches source_zones.domain and zone_summary.zone.

Use left joins. Some graph endpoints and frontier hosts have no hosts row, because references reach beyond the probed inventory. An inner join drops them silently and biases the result toward hosts the crawler happened to probe; quality-report.json counts them under target_endpoint_missing_groups, source_endpoint_missing_groups, and page_hosts_missing_from_sites.

Do not join on `snapshot_row_id`. It appears in every source_* subset, which makes it look like a shared key. In fact it holds each row's SQLite rowid within its own table, so equal values in two subsets point at unrelated records. Its purpose is to give every source row a stable identity, which matters because sanitized url values collide.

Keys, types, and interpretation

hosts.fqdn joins to source_sites.fqdn, DNS fqdn, graph source_domain / target_domain, and frontier host. All of those columns hold hostnames, which may sit several labels below the registrable domain. Some graph endpoints and frontier hosts lack a sites row; they are retained and counted. A link row records its target URL alone, so the page that carried the link stays unknown. The crawler's original link primary key was (source_domain, target_domain, url), without kind; older stored kinds may reflect earlier crawler behavior.

Every source row carries snapshot_row_id, unique within that source table and snapshot. Sanitized URLs repeat: different original query strings collapse onto one URL, so url will not key a row. Source rows stay separate regardless, and graph aggregates report original record counts alongside distinct sanitized URL counts.

Boolean flags use Parquet booleans. DNS and TCP -1 unknowns, numeric -1 sentinels, and empty text become null. A recorded false stays false, and a missing DNS row shows up as has_dns_observation=false, which keeps "we asked and got a negative" separate from "we never asked". DNS and HTTP observations may differ in time and outcome; each is reported on its own terms. UTC observation timestamps use timestamp[us, UTC]. Recovery restored_at has no source timezone and remains a timezone-naive timestamp[us].

See DATA_DICTIONARY.md for every column and schemas.json for machine-readable schemas.

Collection and limitations

The crawler enumerates wordlist-derived names under configured .pl zones, probes DNS and HTTP(S), and expands through discovered web references. Optional Certificate Transparency discovery and historical imported rows contribute different sampling cohorts. The recorded source, legacy, valid, and frontier_excluded fields help separate them; where provenance is empty, it stays empty.

The crawler discards response bodies. content_len counts the bytes actually read and stops at 65,536, so every page past that cap measures alike. HTTPS certificate validation is disabled during the crawl, so a response over HTTPS establishes only that a handshake completed. Link and page crawling proceeds without consulting robots.txt. All of that describes the collection code; the export itself makes no web requests.

Discovery vocabulary, zone configuration, fetch limits, depth and frontier policy, response failures, and rate limiting all bias the graph and leave it incomplete. A 429 records a rate-limited crawl response. unresolved means the query never came back with an answer, while nxdomain means the name was answered as absent — they carry different evidence. pages.fetched records that the crawler finished with the row, whatever the outcome. Stored zone, registrable-domain, and same-site classifications are historical: same_site compares classified registrable domains, which two unrelated organisations under com.pl will share. Data and code span multiple crawler versions, so today's defaults describe today's code while earlier rows were written under whatever applied then.

Sanitization and provenance

All exported URLs have credentials, query strings, and fragments removed. URL paths survive; malformed or unsupported URLs become null. Invalid UTF-8 bytes in URL text are preserved as percent escapes before sanitization. Invalid UTF-8 in identifiers is pseudonymized, and elsewhere becomes the Unicode replacement character, with the two events counted separately. Email-like identifiers and matching text are replaced by consistent HMAC-derived redacted-* pseudonyms; percent-encoded email-like text is decoded first, and embedded URLs and email-like values inside diagnostics are sanitized too. redacted_fields names the changed source fields on each row. Hostnames, addresses, paths, and server headers otherwise survive as observations. Sanitization targets these specific patterns, so text that identifies someone in some other shape can pass through it.

Every source row remains represented. The private pseudonym key and mapping stay out of this release. Preparation repeats given the same snapshot, pinned dependencies, and private key; a fresh key yields fresh pseudonyms. Source row IDs and deterministic sorting hold row identity and order steady, while exact Parquet bytes may shift with library versions.

manifest.json records source schema, row counts, source fingerprints, tool versions, and transformations counted per field. quality-report.json contains full-scan checks, observation ranges, missing-join findings, and collision counts. SHA256SUMS covers all release files except itself. The sanitized source tables preserve every row and column of the original; the original bytes of redacted fields are gone.

This release contains the data and its documentation; the scripts that built it are not distributed. Configure subsets through the documented Hugging Face YAML interface.