ks48/url-atlas
URL Atlas 257,548,097,528 URLs from 105 web corpora, each kept as its own separately-loadable config, plus the raw source dumps two of them were extracted from. 4.35 TiB across 52,244 files. This is the input side of a URL-compression corpus: every source reduced to its URL column and nothing else. It is deliberately not deduplicated or merged — sources are kept intact and overlapping so you can measure what each one contributes, pick the subset you want, and dedup on your own… See the full description on the dataset page: https://huggingface.co/datasets/ks48/url-atlas.
cc-2021-49: final part
cc-2021-43: final part
cc-2021-31: final part
cc-2014-10: +4 index parts
cc-2013-20: +300 index parts (part 2)
cc-2013-20: +300 index parts
cc-2013-48: +300 index parts (part 2)
cc-2013-48: +300 index parts
cc-2014-10: +296 index parts (part 2)
cc-2014-10: +296 index parts
cc-2014-35: +300 index parts (part 2)
cc-2014-35: +300 index parts
cc-2014-41: +300 index parts (part 2)
cc-2014-41: +300 index parts
cc-2014-49: +300 index parts (part 2)
cc-2014-49: +300 index parts
cc-2014-52: +300 index parts (part 2)
cc-2014-52: +300 index parts
cc-2015-06: +300 index parts (part 2)
cc-2015-06: +300 index parts
cc-2015-11: +300 index parts (part 2)
cc-2015-11: +300 index parts
cc-2015-14: +300 index parts (part 2)
cc-2015-14: +300 index parts
cc-2015-22: +300 index parts (part 2)
cc-2015-22: +300 index parts
cc-2015-27: +300 index parts (part 2)
cc-2015-27: +300 index parts
cc-2015-32: +300 index parts (part 2)
cc-2015-32: +300 index parts
cc-2015-40: +300 index parts (part 2)
cc-2015-40: +300 index parts
cc-2016-07: +300 index parts (part 2)
cc-2016-07: +300 index parts
cc-2016-18: +300 index parts (part 2)
cc-2016-18: +300 index parts
cc-2016-30: +300 index parts (part 2)
cc-2016-30: +300 index parts
cc-2016-40: +1 index parts
cc-2016-36: +300 index parts (part 2)
cc-2016-36: +300 index parts
cc-2016-40: +299 index parts (part 2)
cc-2016-40: +299 index parts
cc-2016-44: +300 index parts (part 2)
cc-2016-44: +300 index parts
cc-2016-50: +300 index parts (part 2)
cc-2016-50: +300 index parts
cc-2017-09: +300 index parts (part 2)
cc-2017-09: +300 index parts
cc-2017-26: +69 index parts