open-domain
ccrawl-recrawl-domains
Common Crawl Domain Recrawl
Live fetches of the home page of every ranked domain in Common Crawl's web graph, rendered to Markdown as they are fetched
What is it?
Common Crawl's web graph ranks domains by how central they are, but it does not tell you what those domains actually serve today. This dataset walks that ranking from the top and fetches each domain's home page now, storing the response as one Parquet row with the body, the headers, the timing and the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-domains.ccrawl-domains
Common Crawl Domain Ranks
Web domains ranked by harmonic centrality and PageRank, ready to prioritize a crawl
What is it?
This dataset is the domain-level ranking from Common Crawl's hyperlink web graph, republished as clean Parquet. Common Crawl builds a graph of which domains link to which, then scores every domain by harmonic centrality and PageRank. A high rank means many other well-connected domains link to it, which is a solid proxy for importance when you… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-domains.Open-domain_Financial_QA
Large-scale Open-domain Financial QA (LOFin)
This repository accompanies the dataset used in the paper:
Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documents
(ACL 2025 Findings)
We introduce a benchmark for open-domain financial question answering over standardized documents, with a focus on multi-document and multi-hop reasoning.
📊 Benchmark Overview
This benchmark targets realistic financial QA tasks… See the full description on the dataset page: https://huggingface.co/datasets/HYdsl/Open-domain_Financial_QA.Tiny-Open-Domain-BooksA tiny example dataset consisting of four books dedicated to the open domain in JSONL format:
Alice in Wonderland - Lewis Caroll
Dracula - Bram Stoker
The Wonderful Wizard of Oz - L. Frank Baum
The Count of Monte Cristo - Alexandre Dumas & Auguste Maquet
All works are open domain, thus this dataset is also dedicated to the open domain.
The dataset has been made to have extremely long context lengths, ideally as close to 2048 at possible without cutting off chunks in strange places. Each… See the full description on the dataset page: https://huggingface.co/datasets/Blackroot/Tiny-Open-Domain-Books.mkqa-open-domainopen-domain-ranks
Linkheft Open Domain Ranks: free domain authority data for 10.3 million domains
Try the paid tool: Linkheft on Apify: score any domain list via API, with 4-month trends. First try costs cents; pay only for results.
An open alternative to proprietary "domain authority" scores. For each of the top 10 million domains on the web (plus every Majestic Million domain) this dataset gives:
column
meaning
domain
registrable domain, lowercase (stripe.com)
cc_rank… See the full description on the dataset page: https://huggingface.co/datasets/shaw276/open-domain-ranks.