lavanyaashri/tokenlens-compression-benchmark

TokenLens Compression Benchmark

A benchmark dataset measuring LLM prompt compression quality across 100 Wikipedia articles in 5 categories, generated using TokenLens.

Dataset Description

This dataset contains compression quality measurements for 100 Wikipedia articles compressed at 9 different ratios (0.1 to 0.9) using extractive embedding-based compression. For each article and compression ratio, the dataset records tokens saved, semantic similarity, ROUGE-L score, and combined quality score.

Key Findings

Across 100 Wikipedia articles and 900 compression measurements:

• —science_people (biographies): avg 85.8% semantic similarity at 38% compression
• —technology articles: avg 82.8% semantic similarity at 38% compression
• —science articles: avg 82.1% semantic similarity at 38% compression
• —history articles: avg 79.2% semantic similarity at 38% compression
• —concepts (philosophy/theory): avg 78.5% semantic similarity at 39% compression

ROUGE-L scores are significantly lower than semantic similarity across all categories, indicating compressed answers correctly paraphrase rather than reproduce exact wording. This suggests extractive compression preserves meaning better than surface-level word overlap metrics suggest.

Dataset Structure

900 rows total. Each row represents one Wikipedia article compressed at one ratio.

Compression Method

Extractive compression using sentence-transformers (all-MiniLM-L6-v2). Each document is split into overlapping 200-word chunks. Chunks are ranked by cosine similarity to the evaluation question. The top k% most relevant chunks are kept based on the compression ratio.

Quality is measured by asking the same question to both the full document and the compressed document using Llama 3.2 via Ollama, then comparing answers using ROUGE-L and semantic similarity.

Dataset Statistics

• —Total articles: 100
• —Articles per category: 20
• —Compression ratios tested: 9 (0.1 to 0.9)
• —Total rows: 900
• —Errors: 0
• —LLM used for evaluation: Llama 3.2
• —Embedding model: all-MiniLM-L6-v2

How to Use

from datasets import load_dataset

dataset = load_dataset("lavanyaashri/tokenlens-compression-benchmark")
df = dataset["train"].to_pandas()

# Average semantic similarity by category
df.groupby("category")["semantic_similarity"].mean()

# Average quality by compression ratio
df.groupby("compression_ratio")["combined_score"].mean()

# Find best compression ratio per category
df.groupby(["category", "compression_ratio"])["semantic_similarity"].mean().unstack()

Generation

Generated using TokenLens running Llama 3.2 via Ollama on a local Mac. Evaluation runs in parallel using ThreadPoolExecutor for efficiency. Total generation time approximately 6 hours for 100 articles.

Citation

@misc{tokenlens2026,
  author = {Lavanya Ashri},
  title = {TokenLens Compression Benchmark},
  year = {2026},
  publisher = {HuggingFace},
  url = {https://huggingface.co/datasets/lavanyaashri/tokenlens-compression-benchmark}
}

Links

• —GitHub: https://github.com/lavanyaashri/TokenLens
• —Tool: TokenLens — LLM prompt compression research tool