ks48/url-atlas

URL Atlas 257,548,097,528 URLs from 105 web corpora, each kept as its own separately-loadable config, plus the raw source dumps two of them were extracted from. 4.35 TiB across 52,244 files. This is the input side of a URL-compression corpus: every source reduced to its URL column and nothing else. It is deliberately not deduplicated or merged — sources are kept intact and overlapping so you can measure what each one contributes, pick the subset you want, and dedup on your own… See the full description on the dataset page: https://huggingface.co/datasets/ks48/url-atlas.