ks46/urls-sampled
URLs (hash-sampled) The same 74,918,894,107 URLs as ks46/urls, partitioned by xxh3_64 range into 2,048 chunks of ≈36.6 M rows instead of by SURT key range. Each chunk is a uniform random sample of the whole corpus, and a URL's chunk depends on nothing but the URL itself. Why this exists The SURT layout groups the web by host: shard 1,000 is a contiguous slice of the key space, so it holds whole sites and nothing about any other site. That is what you want for… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-sampled.
Add files using upload-large-folder tool
card: fix strikethrough from bare tildes
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
Add files using upload-large-folder tool
initial commit