SlayerLab/tokenizers
SlayerLab Tokenizers Normalized tokenizer artifacts collected from the contributor directories in slayerlabs/tokenizer, pinned to source commit 1a5cd2c2e4df2287b4c19b3dbf5051f5d460fdc1. The dataset contains one row per tokenizer: the 38 workshop submissions plus the canonical SlayerLab Polish 32k tokenizer by kacperwikiel. Use the Dataset Viewer to sort, filter, and compare tokenizers without navigating folders. Columns author: contributor's exact GitHub username… See the full description on the dataset page: https://huggingface.co/datasets/SlayerLab/tokenizers.
Add fair provisional tokenizer leaderboard
Add multi-agent tokenizer judging benchmark
Add canonical kacperwikiel tokenizer
Add quick tokenizer judging columns
Use GitHub usernames for tokenizer authors
Normalize tokenizers into Parquet table
Add contributor tokenizer artifacts and manifest
initial commit