ks46/urls-sampled

URLs (hash-sampled)

The same 74,918,894,107 URLs as ks46/urls, partitioned by xxh3_64 range into 2,048 chunks of ≈36.6 M rows instead of by SURT key range.

Each chunk is a uniform random sample of the whole corpus, and a URL's chunk depends on nothing but the URL itself.

Why this exists

The SURT layout groups the web by host: shard 1,000 is a contiguous slice of the key space, so it holds whole sites and nothing about any other site. That is what you want for lookups and for compression, and it is what makes that layout 665 GiB instead of this one's 1.79 TB.

It is the wrong shape for three things:

• —A sample. Any prefix of the SURT corpus is a biased slice of the web — alphabetically early hosts, and (because junk sorts first) a concentration of malformed URLs in the early shards. One chunk here is an unbiased 1-in-2,048 sample of everything.
• —Stability. A SURT shard's membership depends on where the 2,047 split points fall, which depends on the corpus they were derived from. Here chunk(url) is a pure function of the URL, so adding or removing corpus data never moves an existing URL between chunks. A train/eval split built on these chunks survives the corpus growing.
• —Parallelism without skew. SURT shards range from 5.6 M to 118.8 M rows because hosts are not evenly sized. These are 36,574,592 to 36,586,653 — a spread of 0.03%.

How a chunk is chosen

chunk(url) = floor( xxh3_64(url) × 2048 / 2⁶⁴ )

Contiguous ranges of the hash space, not hash % 2048. A range stays expressible as a filter, so chunks can be split or merged later without rehashing anything, and each chunk publishes its own [lo, hi) bounds — which is what makes the placement check below possible.

Implementation note, because it is easy to get wrong: the multiply is done in 128 bits. In 64 bits h × 2048 overflows for 99.95% of the hash space and still returns a plausible-looking bucket, because wrapping is itself roughly uniform — 2,048 non-empty chunks of similar size, almost every URL misplaced. The mapping was verified against arbitrary-precision arithmetic at all 2,049 range boundaries, and the shipped chunks were then verified against an independent implementation of xxh3_64 (see below).

Cost of the tradeoff

The same URLs, measured in both layouts:

2.5× the disk for the same data. The reason is in the last two rows: a SURT shard holds ≈190 K hosts densely, so consecutive rows are usually the same site and share long prefixes, which is exactly what DELTA_LENGTH_BYTE_ARRAY and zstd exploit. A hash chunk touches 12.6 M hosts thinly — with ≈491 M hosts in the corpus, most appear in a given chunk once or not at all — so sorting a set of mutually unrelated URLs buys very little. Sorting the chunks at all recovered only 9% over leaving them in arrival order.

Chunks are still sorted by surt(url), url internally, so a chunk is scannable in host order and binary-searchable; it just has no locality to compress.

Layout

data/part-00000.parquet … part-02047.parquet    one column: url (string)
chunks.json                                     per chunk: [lo, hi) hash range, rows, bytes
stats/                                          per-chunk statistics
removed.parquet                                 every URL dropped by validation, with its reason

Parquet V2, zstd level 9, 1,000,000-row row groups.

Verification

Every chunk passes, checked over all 74.9 B rows:

• —placement — each URL's xxh3_64 recomputed from the shipped bytes falls inside its chunk's declared range
• —order — the chunk is sorted by surt(url), url
• —distinct — no duplicate URLs
• —unkeyed — every row has a SURT key

There is deliberately no cross-file sequence check: chunks are ranges of hash space, not key space, so they overlap in SURT order by design.

Placement is checked with the `hashfuncs` DuckDB extension rather than the Rust implementation that did the partitioning. That independence is the point — a verifier calling the same library as the partitioner would confirm its own bug. The two agreed on 109,742,039 URLs across three chunks before it was relied on.

Statistics

stats/ carries the per-chunk table: rows, bytes, total URL length, length percentiles, scheme/query/fragment/port counts, first and last URL.

Corpus-wide tables — hosts, registrable domains, public suffixes, TLDs, length histogram, query keys and values — are not recomputed here. Both datasets hold identical URLs, so they are identical tables, and they are published in ks46/urls.

One caveat if you use the per-chunk numbers: `hosts` is not additive. A host with k URLs appears in 2048 × (1 − (1 − 1/2048)^k) chunks — about 183 of them at the corpus mean of 192 URLs per host, and effectively all 2,048 for any host with more than ≈15,000 URLs. Summing per-chunk host counts gives Σ_hosts chunks(host), not the 490,966,709 distinct hosts. Row counts, byte counts, URL-character totals and the scheme/query/fragment counts are additive.

Provenance and validation

Identical to ks46/urls: 62 URL datasets plus a 43-crawl Common Crawl sweep, globally deduplicated, lightly normalized, and validated with the same url_valid() predicate that removed 5,188,960 rows. removed.parquet here carries the same url + reason + shard record.

The two datasets are cross-checked against each other after every rebuild: URLs, total URL characters, https, www, query, fragment, percent-encoded, non-ASCII, punycode, IP-literal, port and & counts must match exactly across both layouts. Only the layout-dependent figures — bytes on disk, prefix sharing, hosts per file — are allowed to differ.

Usage

from datasets import load_dataset

# one chunk is a uniform 1-in-2048 sample of the whole corpus
import pandas as pd
df = pd.read_parquet("hf://datasets/ks46/urls-sampled/data/part-00000.parquet")

# a reproducible 1% sample: 20 chunks, no scan, no shuffle
files = [f"hf://datasets/ks46/urls-sampled/data/part-{i:05d}.parquet"
         for i in range(20)]

# which chunk holds a given URL — no lookup needed
import xxhash
chunk = (xxhash.xxh3_64_intdigest(url.encode()) * 2048) >> 64

Licensing and attribution

The URLs are collected from public datasets, each under its own license, and from Common Crawl. A URL is a factual reference to a public resource; this dataset contains no page content. Redistribution here does not change any source's terms. Removal requests: open a discussion on this repository.

Common Crawl data is provided under the Common Crawl Terms of Use.