faraa2m/llm-tokens-atlas

Dataset Card for LLM Tokens Atlas llm-tokens-atlas is an open, reproducible benchmark of LLM tokenization across 5 providers (Anthropic claude-opus-4-7, Google gemini-2.5-pro, OpenAI gpt-4o, Mistral mistral-large-latest, Cohere command-r-08-2024) across 5 prompt formats (Markdown, XML, JSON, YAML, Plain text), evaluated on 12,500 real-world prompt requests (n=2,500 per provider, 500 unique prompts × 5 formats × 5 providers). For each (prompt, provider, model, format) cell we… See the full description on the dataset page: https://huggingface.co/datasets/faraa2m/llm-tokens-atlas.