datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
urls-tokenized
URLs (tokenized)
ks46/urls-sampled run through a byte-level
BPE built for URLs, stored as flat uint16 token streams that memory-map
directly into a training loop.
Shards
512
URLs
18,729,786,698
Tokens
664,731,047,208
Vocabulary
8,192
Token dtype
uint16, little-endian
There is no parquet here and the dataset viewer will not render it. These
are raw token bins; see Reading the data below.
Layout
tokenizer/ the exact vocabulary… See the full description on the dataset page: https://huggingface.co/datasets/ks46/urls-tokenized.paper-url-extraction-v1
Papers With Code URL Extraction
A representative dataset for training and evaluating tool-using agents that
find the official GitHub repository and project page for an AI research paper.
It was prepared for the pwc-url-extraction-v1 Prime/verifiers environment.
Splits
Split
Rows
train
4,000
validation
500
test
500
Rows were sampled with seed 13 from up to 120,000 Papers With Code candidates,
stratified by paper year and known URL state.… See the full description on the dataset page: https://huggingface.co/datasets/nielsr/paper-url-extraction-v1.