CoolFace
Datasetpublic

sskapci/domain-intelligence-dataset

Domain Intelligence Dataset A large-scale, derived snapshot of the public internet's domain graph: who links to whom, where domains resolve, which nameservers host them, how their DNS records change over time, and computed authority/spam signals on top. Built from three public sources: ICANN CZDS zone files — daily TLD zone snapshots (.com, .net, .org, …) giving the authoritative set of registered domains and their nameserver delegations. CommonCrawl WARC archives — parsed… See the full description on the dataset page: https://huggingface.co/datasets/sskapci/domain-intelligence-dataset.

sourceHugging Facecc-by-4.0updated 9d agoView on Hugging Face
1likes484downloads
Dataset Card

Domain Intelligence Dataset

A large-scale, derived snapshot of the public internet's domain graph: who links to whom, where domains resolve, which nameservers host them, how their DNS records change over time, and computed authority/spam signals on top.

Built from three public sources:

  1. 1.ICANN CZDS zone files — daily TLD zone snapshots (.com, .net, .org, …) giving the authoritative set of registered domains and their nameserver delegations.
  2. 2.CommonCrawl WARC archives — parsed hyperlinks between pages, aggregated into a domain-to-domain backlink graph (5.1 billion edges).
  3. 3.Active DNS resolution — the pipeline resolved domains directly to capture live A/NS records and their history.

On top of the raw facts, the dataset includes derived analytics: PageRank/TrustRank-style authority scores and PBN (private-blog-network) / link-farm detection signals.

TLD coverage

The domain, zone, and DNS data is centered on four generic TLDs — `.com`, `.net`, `.org`, and `.xyz` — which were the primary ingest targets. The bulk of domains and nameserver/zone history come from these. The backlink graph (domain_outgoing_link) references domains across many other TLDs as link sources/targets, but the registered-domain and zone/DNS coverage is concentrated on .com/.net/.org/.xyz.

No operational or personal data. This release contains only factual and derived web data. It intentionally excludes all business/operational tables — purchase decisions, candidate lists, credentials, user accounts, and internal work queues were never exported.

Scale

MetricValue
Backlink edges (domain→domain, page-level)5.14 billion
Distinct domains212 million
Nameserver-delegation snapshots577 million
DNS/zone historical events~130 million
Uncompressed source volume~1 TiB

Files / Tables

Each table is stored as Parquet (one directory per table; large tables are split into multiple parts). Snappy/zstd compressed.

Core graph

TableRowsWhat it is
domain_outgoing_link5,144,439,407The backlink graph. One row per hyperlink: source_domain → target_domain, with source/target URL and rel attribute. Page-level. This is the headline dataset.
dim_domain212,032,972Domain dimension — the master list of every domain seen, keyed by an internal domain_key.

DNS & nameserver state

TableRowsWhat it is
domain_current_state200,043,012Latest known state per domain (present/dropped in zone, last seen).
domain_current_ns245,185,241Current nameserver set per domain.
domain_snapshot234,907,828Per-day domain presence snapshots from zone files.
domain_ns_snapshot577,137,291Per-day domain→nameserver delegation snapshots.
current_dns_resolution9,899,339Latest resolved DNS records (A/NS) from active resolution.
ns_domain_count3,784,195How many domains each nameserver hosts (concentration signal).
dim_nameserver2,297,025Nameserver dimension.
dim_ip_address969,081IP-address dimension with geolocation.

History (event logs)

TableRowsWhat it is
fact_domain_ns_change104,849,997Every observed nameserver change per domain, timestamped.
fact_domain_event23,546,726Domain lifecycle events (registered / dropped / re-registered).
fact_dns_resolution16,800,376Historical DNS-resolution results.
Note on history depth: the event tables (fact_domain_event, fact_domain_ns_change) begin in August 2026. Earlier months were not retained.

Derived analytics

TableRowsWhat it is
domain_authority15,110,466Computed authority metrics (PageRank / TrustRank / anti-TrustRank-style scores, and DR/CF-style link metrics).
domain_pbn_score5,670,725Private-blog-network / link-farm likelihood scores per domain.
pbn_ns_signal176,316Nameserver-level PBN signals (shared-host / footprint clustering).
domain_keyword1,900,824Keywords associated with domains.

Small reference dimensions

dim_tld, dim_date, dim_ns_provider_pattern, seo_backlink_summary, tld_daily_summary — lookup/summary tables.

Loading

With pandas / pyarrow:

python
import pandas as pd
# one table
df = pd.read_parquet("hf://datasets/sskapci/domain-intelligence-dataset/domain_authority")

With datasets:

python
from datasets import load_dataset
# pick a config by table name
ds = load_dataset("sskapci/domain-intelligence-dataset", "dim_domain", split="train")

The full domain_outgoing_link graph is 5B+ rows — stream it or read part files individually rather than loading whole:

python
import pyarrow.dataset as ds
graph = ds.dataset("hf://datasets/sskapci/domain-intelligence-dataset/domain_outgoing_link", format="parquet")

Provenance & method

  • —Zone files: ICANN Centralized Zone Data Service (CZDS), ingested daily; diffs computed to detect registrations and drops.
  • —Backlinks: parsed from CommonCrawl WARC archives, aggregated to the domain level.
  • —DNS: active resolution of domains for live A/NS records.
  • —Authority: personalized-PageRank / TrustRank family algorithms over the link graph.
  • —PBN detection: out-degree, source-overlap, and shared-nameserver footprint analysis.

Uses

  • —Web-graph and link-analysis research
  • —SEO / backlink-profile study
  • —Domain reputation, spam / link-farm detection
  • —DNS / hosting-concentration analysis
  • —Historical domain lifecycle study

License

Released under CC-BY-4.0. The underlying facts come from public web sources (CommonCrawl, ICANN CZDS zone data, public DNS). Attribute this dataset if you use it.

Limitations

  • —A point-in-time snapshot; not continuously updated.
  • —Backlink coverage is bounded by CommonCrawl's crawl scope.
  • —Derived scores (authority, PBN) are heuristic, not ground truth.
  • —Event history begins August 2026.