Datasetpublic

metehan777/common-crawl-2026-21

Common Crawl SEO & AEO/GEO Dataset — CC-MAIN-2026-21 Web pages from the May 2026 Common Crawl (CC-MAIN-2026-21) that mention SEO, Answer Engine Optimization (AEO), or Generative Engine Optimization (GEO) — filtered to English content and ranked by Common Crawl Web Graph harmonic centrality and PageRank. 🔎 Interactive Explorer A Gradio app to search, filter, and analyze this dataset: https://huggingface.co/spaces/metehan777/cc-seo-explorer Tabs: SEO raw search ·… See the full description on the dataset page: https://huggingface.co/datasets/metehan777/common-crawl-2026-21.

sourceHugging Facecc0-1.0updated 3mo agoView on Hugging Face

0likes522downloads

Dataset Card

Common Crawl SEO & AEO/GEO Dataset — CC-MAIN-2026-21

Web pages from the May 2026 Common Crawl (CC-MAIN-2026-21) that mention SEO, Answer Engine Optimization (AEO), or Generative Engine Optimization (GEO) — filtered to English content and ranked by Common Crawl Web Graph harmonic centrality and PageRank.

🔎 Interactive Explorer

A Gradio app to search, filter, and analyze this dataset: https://huggingface.co/spaces/metehan777/cc-seo-explorer

Tabs: SEO raw search · AEO+GEO combined search (full-text) · N-gram analysis · Domain leaderboard · Distinctive terms · Topic clusters · SEO filtered.

Files

SEO

File	Rows	Description
`seo_CC-MAIN-2026-21_raw.parquet`	4.6M	All English pages with exact word "seo" — snippet (500c)
`seo_CC-MAIN-2026-21_filtered.parquet`	~1.3M	Quality-filtered SEO + domain scores
`seo_emb_filtered_part00{0-3}.parquet`	~1.3M	MiniLM embeddings (384-dim, float16) of filtered SEO

Answer Engine Optimization (AEO)

File	Rows	Description
`answer_engine_optimization_CC-MAIN-2026-21_raw.parquet`	143K	Raw matches, snippet
`answer_engine_optimization_CC-MAIN-2026-21_full_text.parquet`	130K	Raw, text up to 4000c
`answer_engine_optimization_CC-MAIN-2026-21_quality_fulltext.parquet`	58K	Quality-filtered, full page text (~14.7K chars avg)
`answer_engine_optimization_final.json`	58K	Quality-filtered + domain scores + composite
`answer_engine_optimization_embeddings.parquet`	58K	MiniLM embeddings (384-dim)
`answer_engine_optimization_ngrams.json`	—	N-gram analysis (49.9M tokens)
`answer_engine_optimization_clusters.json`	20	Auto-named semantic topic clusters

Generative Engine Optimization (GEO)

Same file set as AEO (234K raw / 213K full text / ~75K quality, 79M tokens).

Cross-corpus analysis

File	Description
`discriminative_terms.json`	Log-odds ratio (Monroe et al. 2008) — words distinguishing AEO vs GEO vs SEO

Schema

Raw / snippet parquet

url          (string)  — Original page URL
snippet      (string)  — First 500 chars of extracted text
warc         (string)  — Source WET file name
crawl_date   (string)  — WARC crawl timestamp (ISO 8601)
hit_count    (int64)   — Exact term matches in the page

Quality-filtered (JSON / quality_fulltext parquet)

Additional fields:

text         (string)  — Full extracted page text (quality_fulltext only)
domain       (string)  — Registered domain (eTLD+1, via tldextract)
hc           (float)   — Harmonic Centrality (CC Web Graph 2026 mar-apr-may)
pr           (float)   — PageRank (CC Web Graph 2026 mar-apr-may)
q_score      (float)   — Text quality score (FineWeb + Gopher + C4)
composite    (float)   — 0.5×log_norm(HC) + 0.3×q_score + 0.2×min(hits/20,1)
n_words      (int)     — Word count

Embeddings parquet

url, domain  (string)
e0 … e383    (float16) — sentence-transformers/all-MiniLM-L6-v2, normalized

Key Findings

Term prevalence (per 100K WET files, English):

—SEO: extremely common (~176 matches/file)
—GEO: 234,358 raw matches
—AEO: 143,327 raw matches → GEO is ~60% more discussed than AEO

Distinctive vocabulary (log-odds):

—AEO ↔ customer, hubspot, software, sales, platform, premium (B2B SaaS framing)
—GEO ↔ geo, generative, chatgpt, audio, strategy, brand (generative-AI framing)
—SEO ↔ wordpress, blog, plugin, menu, login (classic web/blog)

Quality Filtering

All quality-filtered files apply the following pipeline (inspired by FineWeb, Gopher, and C4):

—Language: CC's WARC-Identified-Content-Language = eng
—Word count: 50–100,000 words
—Mean word length: 3–10 characters
—Alphabet ratio: > 50%
—Bullet fraction: < 80% of lines
—Terminal punctuation: > 15% of lines end with .!?"'
—Symbol density: < 8%
—Repetition: > 15% unique word ratio
—Junk phrases: < 3 cookie/spam phrases
—Navigation pages: < 70% short lines (< 3 words)
—Free-hosting blocklist: blogspot, wordpress.com, wixsite, etc.

Domain Ranking

Domain scores come from the CC Web Graph cc-main-2026-mar-apr-may, top 5M domains by harmonic centrality.

Source

—Crawl: CC-MAIN-2026-21 (May 8–21, 2026)
—100,000 WET files, 2.16 billion web pages total
—Processing: custom Python pipeline on vast.ai 4x NVIDIA B200