datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
fineweb-tokenized
FineWeb Tokenized
> 4 trillion tokens of the pre-tokenized data the 🌐 web has to offer
What is it?
This is a pre-tokenized version of the HuggingFaceFW/fineweb dataset (currently in-progress, tokenization of the ~15 trillion tokens corpus is ongoing). The data is being pre-processed and tokenized using the AnisoleAI BPE tokenizer (52,022 vocabulary size) and packed into compact uint16 Parquet shards.
By distributing the pre-tokenized corpus, we eliminate… See the full description on the dataset page: https://huggingface.co/datasets/anisoleai/fineweb-tokenized.tokenizers-dependents
tokenizers metrics
This dataset contains metrics about the huggingface/tokenizers package.
Number of repositories in the dataset: 11460
Number of packages in the dataset: 124
Package dependents
This contains the data available in the used-by
tab on GitHub.
Package & Repository star count
This section shows the package and repository star count, individually.
Package
Repository
There are 14 packages that have more than 1000 stars.
There are 41… See the full description on the dataset page: https://huggingface.co/datasets/open-source-metrics/tokenizers-dependents.seamless-align-enA-jaA.tokenized.encodecfineweb-tokenized-fake
What is it?
It's similar to anisolai/fineweb-tokenized but fake.
I don't understand why I did that :)
WARNING:
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN SOMEONE.
WHY ARE YOU DOWNLOADING IT? YOU COULD LOSE MILLIONS OF DOLLARS IF YOU USE IT TO TRAIN… See the full description on the dataset page: https://huggingface.co/datasets/mondk/fineweb-tokenized-fake.seamless-align-enA-hiA.tokenized.encodecloracle-pretrain-v5-qwen14b-tokensRULER-8192-Qwen2.5-3B-tokenizerseamless-align-enA-esA.tokenized.encodecseamless-align-enA-viA.tokenized.encodecseamless-align-deA-enA.tokenized.encodeccarbon-tokenized-corpus
Dataset Summary
AINovice2005/carbon-tokenized-corpus is the tokenized sample of AINovice2005/carbon-cpu-enriched-sequences-sampled .
Schema
The current dataset contains the following fields:
Field
Type
Description
record_id
string
Source/reference sequence identifier
start
int64
Start coordinate of the sequence interval
end
int64
End coordinate of the sequence interval
token_ids
list
Integer token IDs produced by the tokenizer
token_mask
list… See the full description on the dataset page: https://huggingface.co/datasets/AINovice2005/carbon-tokenized-corpus.0399-tv-valid-clean-sft-tokenized-llmjp4-8bToken-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data
Token-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_token_prices_ohlc (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Token-Prices-OHLC-Ethereum-Cryptocurrency-Data.Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data
Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_token_vwap (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
bucket_start
timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Token-VWAP-Ethereum-Cryptocurrency-Data.exp-pool-repository-code-dolma2-tokenized
Locus EXP Repository Code - Dolma 2 tokenized
Pretokenized experiment pool for reproducible proxy-training runs.
MANIFEST.json is the authoritative schema, provenance, checksums, and train/holdout assignment.
shards/shard-NNNNN/tokens.bin stores little-endian int32 token IDs.
offsets.bin stores little-endian int64 document boundaries.
index.parquet stores document IDs, offsets, and compact filter fields.
metadata.parquet stores complete source metadata and is downloaded only… See the full description on the dataset page: https://huggingface.co/datasets/placeholderlabs/exp-pool-repository-code-dolma2-tokenized.tokenized_sampleToken-To-Token-Cross-Pool-VWAP-Ethereum-Cryptocurrency-Data
Token-To-Token-Cross-Pool-VWAP-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_token_cross_pool_vwap (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Token-Cross-Pool-VWAP-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Token-Cross-Pool-VWAP-Ethereum-Cryptocurrency-Data.Ichigo-instruction-tokenized-v0.2seamless-align-enA-frA.tokenized.encodecseamless-align-enA-koA.tokenized.encodecERC20-Tokens-Ethereum-Cryptocurrency-Data
ERC20-Tokens-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB erc20_tokens (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/ERC20-Tokens-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2023-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
block_number
int64
tx_index
int32
contract_id… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/ERC20-Tokens-Ethereum-Cryptocurrency-Data.Token-To-Token-Prices-Swap-Prints-Ethereum-Cryptocurrency-Data
Token-To-Token-Prices-Swap-Prints-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_token_prices_swap_prints (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Token-Prices-Swap-Prints-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2020-05 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Token-Prices-Swap-Prints-Ethereum-Cryptocurrency-Data.c4-token-logprobsRULER-32768-Qwen2.5-3B-tokenizersib200-xlmr-tokenized
SIB-200 Tokenized by XLM-R Large
This repository provides pre-tokenized versions of SIB-200 used in the paper:Cross-Prompt Encoder for Low-Performing LanguagesFindings of IJCNLP–AACL 2025; preprint at arXiv:2508.10352.
The dataset is released to support zero-shot and fully supervised cross-lingual experiments presented in our paper,
ensuring consistent and reproducible tokenization across all languages and experimental settings.
The dataset is organized as a multi-config Hugging… See the full description on the dataset page: https://huggingface.co/datasets/mikaberidze/sib200-xlmr-tokenized.Token-To-Fiat-VWAP-Ethereum-Cryptocurrency-Data
Token-To-Fiat-VWAP-Ethereum-Cryptocurrency-Data
Hive-partitioned Parquet export of BlockDB token_to_fiat_vwap (Ethereum).
Load
from datasets import load_dataset
ds = load_dataset("BlockDB/Token-To-Fiat-VWAP-Ethereum-Cryptocurrency-Data", split="train")
Files live under data/year=YYYY/month=MM/part-NNNN.parquet.
Range: 2015-08 .. 2026-06 (UTC calendar months).
Schema
column
type
block_timestamp
timestamp
bucket_start
timestamp… See the full description on the dataset page: https://huggingface.co/datasets/BlockDB/Token-To-Fiat-VWAP-Ethereum-Cryptocurrency-Data.dclm_10B_tokensmixed-pretraining-tokenizedfinemath-4plus-tokenizedseamless-align-enA-zhA.tokenized.encodec