datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
datasets-tests-compressionLLM_compression_calibration
LLM Compression Calibration dataset
This dataset is the default calibration dataset used by Neural Magic for one-shot compression of Large Language Models (LLMs).
Note: This dataset is the result of active research and subject to change without notice.
Dataset Details
Dataset Sources
The current version of this dataset is compiled from data from these datasets:
garage-bAInd/Open-Platypus: 10,000 samples
Data Fields
The dataset contains 2 data… See the full description on the dataset page: https://huggingface.co/datasets/neuralmagic/LLM_compression_calibration.compression-pretraining-data
Dataset
Each example contains prompt (chat format) and target fields.
from datasets import load_dataset
ds = load_dataset("leonli66/compression-pretraining-data", "<config_name>")
lean-proof-compression
LeanPolish: A Kernel-Verified Dataset and Symbolic Compression Framework for Lean 4 Proofs
A dataset of Lean 4 proof rewrite pairs produced by LeanPolish,
a kernel-verified proof-shortening tool. Every accepted
(original, replacement) pair was kernel-checked under Lean 4.21.0
with Mathlib v4.21.0 before emission, and the rewritten file was
re-elaborated end-to-end by a separate out-of-process verifier.
The dataset is suitable for training models that learn to compress,
simplify… See the full description on the dataset page: https://huggingface.co/datasets/leanpolish-anon/lean-proof-compression.round-trip-code-compressionvision-token-compression-bench
OPTIC-Bench
Optical Text In-Context Benchmark: how reliably do LLMs consume text
delivered as rendered images versus plain text tokens?
In summary, the evaluation reported here finds that optical text compression
is effective only within a narrow and specific envelope. Delivering content
as rendered images genuinely reduces input tokens, by thirteen to
fifty-four per cent depending on the model and the language, but only when
the document is long, the rendering is dense and the… See the full description on the dataset page: https://huggingface.co/datasets/translorentz/vision-token-compression-bench.all-deletion-compressionswikipedia-deletion-compressionssentence-compression
Dataset Card for Sentence Compression
This dataset is a collection of text-simplified pairs from the Sentence Compression project. See Sentence Compression for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "text", "simplified"
Column types: str, str
Examples:{
'text': "The USHL completed an expansion draft on Monday as 10 players who were on the rosters of… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/sentence-compression.sentence-compression
Dataset Card for "sentence-compression"
Dataset Summary
Dataset with pairs of equivalent sentences.
The dataset is provided "AS IS" without any warranty, express or implied.
Google disclaims all liability for any damages, direct or indirect, resulting from using the dataset.
Disclaimer: The team releasing sentence-compression did not upload the dataset to the Hub and did not write a dataset card. These steps were done by the Hugging Face team.
Supported Tasks… See the full description on the dataset page: https://huggingface.co/datasets/embedding-data/sentence-compression.llm-compressionThis is the compression corpora dataset used in the paper "Compression Represents Intelligence Linearly".
We find that LLMs’ intelligence – reflected by benchmark scores – almost linearly correlates with their ability to compress external text corpora. We measure intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning, and provide the corresponding compression corpora here respectively named cc, python, and arxiv_math.
Load the data… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/llm-compression.cai-semantic-equivalence-benchmark
Contradish CAI-Bench
The semantic equivalence benchmark from Contradish
Do AI systems give the same answer when the wording changes but the meaning does not?
Contradish CAI-Bench measures semantic invariance: whether an AI system remains behaviorally consistent across prompts that express the same intent in different words.
This Hugging Face release contains 420 human-readable prompt pairs across 19 domains. Contradish is the official benchmark runner, scoring… See the full description on the dataset page: https://huggingface.co/datasets/compressionawareintelligence/cai-semantic-equivalence-benchmark.compression_test2compression_test3compression_testlace-semantic-compression
LACE — Latent Adaptive Compression Engine
Semantic Compression Under Physical Channel Constraints: Cognitive Phase Transitions Under Bandwidth Constraints
Théophile Lafargue · April 2026 · Patent FR2511116
What this is
198 operational tasks (defense, medical, industrial) used to study what emerges when you force discrete semantic compression under LoRa/SMS physical constraints.
v1 result: Domain clustering real (mean coherence 68.4%). Retrieval/inference separation not… See the full description on the dataset page: https://huggingface.co/datasets/ox-ox/lace-semantic-compression.OpenR1-DeepSeek-R1-Distill-Qwen-7BBlockwiseCompressedReasoniong-Level-3-CompressionRate-0.1compression_dataset
Compression Dataset — ER Training Subset (3,200)
The 3,200 examples selected by the Efficient-Reasoning training script, stored as Parquet for Hugging Face Dataset Viewer support. This is a subset of daman1209arora/compression_dataset, not the full upstream dataset.
Exact selection
run_rloo_deepseek_1.5B_compression.sh sets --max_samples 3200. The loader in openrlhf/utils/utils.py takes the first 3,200 source rows before shuffling with seed 42. This repository… See the full description on the dataset page: https://huggingface.co/datasets/zjhhhh/compression_dataset.mn-context-compression-dataset-v1
MN Context Compression Dataset v1
Author: Homer Quan
This dataset is used to train context-compression models for improving the context efficiency of multi-agent runtimes, especially MirrorNeuron and the broader work at mirrorneuron.io.
We use this dataset to train models such as homerquan/mn-context-engine-lora-v2, and later protected-fact-focused context engines. The data emphasizes exact protected-span retention, source-reference preservation, budget-conditioned compression, and… See the full description on the dataset page: https://huggingface.co/datasets/homerquan/mn-context-compression-dataset-v1.semantic-compression-sft
sematic-compression-sft
Dataset Summary
sematic-compression-sft is a synthetic supervised fine-tuning dataset for semantic compression.
The task is to convert verbose natural-language or code inputs into compact outputs that preserve reasoning-relevant information.
This dataset is designed for training compression models/adapters used before downstream LLM inference to reduce prompt size while retaining functional utility.
Goal
The objective is not generic… See the full description on the dataset page: https://huggingface.co/datasets/Sudhendra/semantic-compression-sft.dolly-15k-prompt-compression
Dolly-15k Prompt Compression
This dataset contains compressed versions of the Databricks Dolly-15k prompts. Each prompt was compressed using the gpt-5-nano model to minimize input tokens while preserving all constraints. You can explore the downstream model that relies on this data in the companion Space: Very Small Prompt Compression Demo.
Compression model: gpt-5-nano
Source dataset: databricks/databricks-dolly-15k
Rows: 15,000
Aggregate token savings: 289,540 → 215,219 tokens… See the full description on the dataset page: https://huggingface.co/datasets/gravitee-io/dolly-15k-prompt-compression.OpenR1-DeepSeek-R1-Distill-Qwen-7BBlockwiseCompressedReasoniong-Level-2-CompressionRate-0.3er_cost_marginrl_r1_distill_1.5b_compression_n16_b512_32k_lr1e-6_kl0_seed42-rollouts
er_cost_marginrl_r1_distill_1.5b_compression_n16_b512_32k_lr1e-6_kl0_seed42 rollouts
This dataset contains one compressed JSONL shard for every completed training
step. The step and rollout_index columns uniquely locate a rollout within
this training run. Run metadata and per-step row counts are recorded in
rollout_manifest.json.
OpenR1-DeepSeek-R1-Distill-Qwen-7BBlockwiseCompressedReasoniong-Level-2-CompressionRate-0.8OpenR1-DeepSeek-R1-Distill-Qwen-7BBlockwiseCompressedReasoniong-Level-3-CompressionRate-0.5Fathom-V0.4-RL-CompressionOpenR1-DeepSeek-R1-Distill-Qwen-7BBlockwiseCompressedReasoniong-Level-3-CompressionRate-0.8Compression_CoCidb-invariant-compression-fidelity-v0.1
What this dataset tests
Whether compression keeps the invariant.
Not just the output.
A student can match answerswhile losing structure.
This benchmark detects that.
Why this exists
Compression can create proxy behavior.
The model learnswhat to saynot what must be preserved.
This set separates:
faithful retention
proxy matching
invariant loss
Data format
Each row contains:
original prompt and compressed prompt
teacher output and student output
an… See the full description on the dataset page: https://huggingface.co/datasets/ClarusC64/idb-invariant-compression-fidelity-v0.1.task1340_msr_text_compression_compression
Dataset Card for Natural Instructions (https://github.com/allenai/natural-instructions) Task: task1340_msr_text_compression_compression
Additional Information
Citation Information
The following paper introduces the corpus in detail. If you use the corpus in published work, please cite it:
@misc{wang2022supernaturalinstructionsgeneralizationdeclarativeinstructions,
title={Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP… See the full description on the dataset page: https://huggingface.co/datasets/Lots-of-LoRAs/task1340_msr_text_compression_compression.