datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
urls-tokenized
URLs (tokenized)
ks46/urls-sampled run through a byte-level
BPE built for URLs, stored as flat uint16 token streams that memory-map
directly into a training loop.
Shards
512
URLs
18,729,786,698
Tokens
664,731,047,208
Vocabulary
8,192
Token dtype
uint16, little-endian
There is no parquet here and the dataset viewer will not render it. These
are raw token bins; see Reading the data below.
Layout
tokenizer/ the exact vocabulary… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-tokenized.malicious_urlsDataset from: bgspaditya/byt-malicious-url-treatment
grokipedia-urls
Grokipedia URLs
This (unofficial) repo hosts all current URLs to Grokipedia articles (incl. last modification timestamp).
At the moment, 885.279 articles are available.
Have fun with them!
service-urlsseed-crawl-urlsURLSumURLSum