datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
urls
URLs
74,918,894,107 deduplicated, validated URLs, sorted by
SURT key
and split into 2,334 range shards.
As plain text the URLs are 5.8 TiB, averaging 84 characters each. Sorted by
SURT key and delta-encoded they fit in 665.6 GiB — 9.54 bytes per URL, a
8.85× reduction. That is the whole point of the ordering: SURT puts URLs
from the same site next to each other, DELTA_LENGTH_BYTE_ARRAY then stores only
where each row differs from the one above it, and zstd compresses what is… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls.fineweb_urls
Dataset Card for fineweb_urls
This dataset provides the URLs and top-level domains associated with training records in HuggingFaceFW/fineweb. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/fineweb_urls.zyda_urls
Dataset Card for zyda_urls
This dataset provides the URLs and top-level domains associated with training records in Zyphra/Zyda. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so, it allows… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/zyda_urls.c4_urls_en.noblocklist
Dataset Card for c4_urls_en.noblocklist
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English no blocklist variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.noblocklist.culturax_urls
Dataset Card for culturax_urls
This dataset provides the URLs and top-level domains associated with training records in uonlp/CulturaX. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so, it… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/culturax_urls.urls-tokenized
URLs (tokenized)
ks46/urls-sampled run through a byte-level
BPE built for URLs, stored as flat uint16 token streams that memory-map
directly into a training loop.
Shards
512
URLs
18,729,786,698
Tokens
664,731,047,208
Vocabulary
8,192
Token dtype
uint16, little-endian
There is no parquet here and the dataset viewer will not render it. These
are raw token bins; see Reading the data below.
Layout
tokenizer/ the exact vocabulary… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-tokenized.ccrawl-recrawl-urls
Common Crawl URL Recrawl
Live refetches of pages from Common Crawl's URL index, with the body inline and the text already extracted
What is it?
Common Crawl publishes which URLs it saw and when, but the page bodies live in WARC archives that are awkward to query and are as old as the crawl that made them. This dataset takes the URL index for a single monthly crawl and fetches the pages again now, storing each response as one Parquet row with the body, the… See the full description on the dataset page: https://huggingface.co/datasets/open-index/ccrawl-recrawl-urls.c4_urls_en
Dataset Card for c4_urls_en
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.c4_urls_multilingual
Dataset Card for c4_urls_multilingual
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (multilingual variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_multilingual.falcon-refinedweb_urls
Dataset Card for falcon-refinedweb_urls
This dataset provides the URLs and top-level domains associated with training records in tiiuae/falcon-refinedweb. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers.… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/falcon-refinedweb_urls.zyda-2_urls
Dataset Card for zyda-2_urls
This dataset provides the URLs and top-level domains associated with training records in Zyphra/Zyda-2. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so, it allows… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/zyda-2_urls.c4_urls_en.noclean
Dataset Card for c4_urls_en.noclean
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (English no clean variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_en.noclean.dclm-baseline-1.0-parquet_urls
Dataset Card for dclm-baseline-1.0-parquet_urls
This dataset provides the URLs and top-level domains associated with training records in mlfoundations/dclm-baseline-1.0-parquet. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/dclm-baseline-1.0-parquet_urls.urls-sampled
URLs (hash-sampled)
The same 74,918,894,107 URLs as
ks46/urls, partitioned by
xxh3_64 range into 2,048 chunks of ≈36.6 M rows instead of by SURT key range.
Each chunk is a uniform random sample of the whole corpus, and a URL's chunk
depends on nothing but the URL itself.
Why this exists
The SURT layout groups the web by host: shard 1,000 is a contiguous slice of the
key space, so it holds whole sites and nothing about any other site. That is
what you want for… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-sampled.ARCHIVE-TEXT-URLS
Internet Archive English Text URLs Dataset
Dataset Description
This dataset contains 11,151,637 direct download URLs to OCR-processed text files from the Internet Archive's digital library. All entries are English-language texts spanning books, documents, historical records, and various other written materials.
Dataset Summary
Total Rows: 11,151,637
Language: English
Source: Internet Archive
Format: CSV with metadata and direct text file URLs
Text… See the full description on the dataset page: https://huggingface.co/datasets/Navanjana/ARCHIVE-TEXT-URLS.madlad-400_urls_noisy
Dataset Card for madlad-400_urls_noisy
This dataset provides the URLs and top-level domains associated with training records in allenai/MADLAD-400 (noisy variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/madlad-400_urls_noisy.madlad-400_urls_clean
Dataset Card for madlad-400_urls_clean
This dataset provides the URLs and top-level domains associated with training records in allenai/MADLAD-400 (clean variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/madlad-400_urls_clean.fineweb-edu_urls
Dataset Card for fineweb-edu_urls
This dataset provides the URLs and top-level domains associated with training records in HuggingFaceFW/fineweb-edu. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/fineweb-edu_urls.fineweb-2_urls
Dataset Card for fineweb-2_urls
This dataset provides the URLs and top-level domains associated with training records in HuggingFaceFW/fineweb-2. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/fineweb-2_urls.onlysports_dataset_urls
Dataset Card for onlysports_dataset_urls
This dataset provides the URLs and top-level domains associated with training records in Chrisneverdie/OnlySports_Dataset. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/onlysports_dataset_urls.dclm-dedup_urls
Dataset Card for dclm-dedup_urls
This dataset provides the URLs and top-level domains associated with training records in Zyphra/dclm-dedup. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/dclm-dedup_urls.c4_urls_realnewslike
Dataset Card for c4_urls_realnewslike
This dataset provides the URLs and top-level domains associated with training records in allenai/c4 (realnewslike variant). It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4_urls_realnewslike.infimm-webmath-40b_urls
Dataset Card for infimm-webmath-40b_urls
This dataset provides the URLs and top-level domains associated with training records in Infi-MM/InfiMM-WebMath-40B. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/infimm-webmath-40b_urls.moscar_urls
Dataset Card for moscar_urls
This dataset provides the URLs and top-level domains associated with training records in oscar-corpus/mOSCAR. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing so, it… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/moscar_urls.101_billion_arabic_words_dataset_urls
Dataset Card for 101_billion_arabic_words_dataset_urls
This dataset provides the URLs and top-level domains associated with training records in ClusterlabAi/101_billion_arabic_words_dataset. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/101_billion_arabic_words_dataset_urls.c4-chinese-zhtw_urls
Dataset Card for c4-chinese-zhtw_urls
This dataset provides the URLs and top-level domains associated with training records in erhwenkuo/c4-chinese-zhtw. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers.… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/c4-chinese-zhtw_urls.mc4-es-sampled_urls
Dataset Card for mc4-es-sampled_urls
This dataset provides the URLs and top-level domains associated with training records in bertin-project/mc4-es-sampled. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/mc4-es-sampled_urls.clean_mc4_it_urls
Dataset Card for clean_mc4_it_urls
This dataset provides the URLs and top-level domains associated with training records in gsarti/clean_mc4_it. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only those record identifiers. In doing… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/clean_mc4_it_urls.