datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
datasets-tests-compressionLLM_compression_calibration
LLM Compression Calibration dataset
This dataset is the default calibration dataset used by Neural Magic for one-shot compression of Large Language Models (LLMs).
Note: This dataset is the result of active research and subject to change without notice.
Dataset Details
Dataset Sources
The current version of this dataset is compiled from data from these datasets:
garage-bAInd/Open-Platypus: 10,000 samples
Data Fields
The dataset contains 2 data… See the full description on the dataset page: https://huggingface.co/datasets/neuralmagic/LLM_compression_calibration.comma_video_compression_challenge_pr_archive
comma video compression challenge - PR archive corpus
Card last refreshed: 2026-05-11 (companion research artifacts section
added).
This dataset captures every scored Pull Request submitted to
commaai/comma_video_compression_challenge,
the public 2026 contest to compress comma's 0.mkv reference dashcam video
under perceptual + temporal scorer constraints.
For each scored PR we publish:
archive.zip - the exact compressed-archive bytes that were scored
by the contest evaluation… See the full description on the dataset page: https://huggingface.co/datasets/adpena/comma_video_compression_challenge_pr_archive.codecpilot-compression-decision-dataset
CodecPilot Compression-Decision Dataset
中文简介：这是用于训练和评估“图像压缩参数决策器”的多源数据集。模型的目标不是改变输入格式，而是在保持输入格式不变的前提下，根据图像内容和用户偏好（均衡压缩、极致压缩、尽量原图）预测合适的编码参数，减少反复试编码的成本。
The repository contains immutable image inputs, compression-decision labels, candidate records, manifests, integrity data, and dataset-preparation reports. Images are packaged without re-encoding so the original compressed bytes are preserved.
数据集组件 / Dataset components
The repository intentionally keeps two… See the full description on the dataset page: https://huggingface.co/datasets/winrisef/codecpilot-compression-decision-dataset.fastwam-lerobotImageNet-C-jpeg_compression-severity_5compression-pretraining-data
Dataset
Each example contains prompt (chat format) and target fields.
from datasets import load_dataset
ds = load_dataset("leonli66/compression-pretraining-data", "<config_name>")
compression-drives-truth
Compression Drives Truth — Evaluation & Reproduction Kit
Paired evaluation data and reproduction scripts for the paper
"Compression Favors Consistency, Not Truth"
(arXiv:2603.11749).
What's Inside
eval/ — Paired Evaluation Sets
The core artifact. Each JSONL file contains pairs of (correct, incorrect) completions
for the same problem prompt. A model shows "truth bias" if it assigns lower NLL
to correct completions on average.
File
Description
N pairs… See the full description on the dataset page: https://huggingface.co/datasets/krestnikov/compression-drives-truth.cs527-optical-compression-trajectories
CS527 Optical Compression Agent Trajectories
Agent trajectories from the paper "Optical Compression for Agentic Code Understanding" (CS 527 Group-9, UIUC).
Overview
This dataset contains 200 agent trajectories (100 per condition) from evaluating optical compression on SWE-bench Verified using GPT-5-mini and mini-swe-agent.
Text condition: Standard text-based agent (all tool outputs as plain text)
Optical condition: Code-heavy tool outputs rendered as monospace images… See the full description on the dataset page: https://huggingface.co/datasets/autoprogrammer/cs527-optical-compression-trajectories.TPMS_CompressionTestscorruption-jpeg_compression
Corruption Dataset: Jpeg_Compression
Dataset Description
This dataset contains corrupted versions of ImageNet-1K images using jpeg_compression corruption. It is part of the ImageNet-C benchmark for evaluating model robustness to common image corruptions.
Dataset Structure
Train: 1,281,167 corrupted images
Validation: 50,000 corrupted images
Classes: 1000 ImageNet-1K classes
Format: Arrow (Hugging Face Datasets)
Corruption Type: Jpeg_Compression… See the full description on the dataset page: https://huggingface.co/datasets/MarMaster/corruption-jpeg_compression.lean-proof-compression
LeanPolish: A Kernel-Verified Dataset and Symbolic Compression Framework for Lean 4 Proofs
A dataset of Lean 4 proof rewrite pairs produced by LeanPolish,
a kernel-verified proof-shortening tool. Every accepted
(original, replacement) pair was kernel-checked under Lean 4.21.0
with Mathlib v4.21.0 before emission, and the rewritten file was
re-elaborated end-to-end by a separate out-of-process verifier.
The dataset is suitable for training models that learn to compress,
simplify… See the full description on the dataset page: https://huggingface.co/datasets/leanpolish-anon/lean-proof-compression.round-trip-code-compressionmsr_text_compressionThis dataset contains sentences and short paragraphs with corresponding shorter (compressed) versions. There are up to five compressions for each input text, together with quality judgements of their meaning preservation and grammaticality. The dataset is derived using source texts from the Open American National Corpus (ww.anc.org) and crowd-sourcing.compression-eval-datasets
Compression Evaluation Datasets
Common test sets for learned image compression evaluation, packaged for easy download.
Contents
Folder
Description
Images
Size
kodak/
Kodak PhotoCD (kodim01–kodim24)
24
~15MB
tecnick/
Tecnick RGB test images (1200×1200)
40
~66MB
clic2021_valid/
CLIC professional validation images (local folder name: CLIC2021_valid)
41
~129MB
Download
# Full dataset
hf download SCU-VIP-Lab/compression-eval-datasets… See the full description on the dataset page: https://huggingface.co/datasets/SCU-VIP-Lab/compression-eval-datasets.svs-lame-compression-jpeg-vs-neural
Comparaison visuelle : compression neuronale vs JPEG sur lames histopathologiques
Ce dataset permet à un anatomopathologiste de juger à l'œil nu si une image
de lame numérique compressée par un réseau de neurones est visuellement
équivalente à la même lame compressée en JPEG (qui est le standard)
En une phrase
On a pris 5 lames histopathologiques au format SVS, on les a compressées avec
4 modèles neuronaux et avec JPEG Q75, à
deux niveaux d'agressivité (q5 ≈… See the full description on the dataset page: https://huggingface.co/datasets/nathbns/svs-lame-compression-jpeg-vs-neural.Parameter-Golf-V17-512Cube-XYZ-Priority-Orbital-Compression
Parameter Golf V17 — 512Cube Solution Bank
This is an English research-control dataset for OpenAI Parameter Golf work. It is not a replacement for FineWeb and must not be used as a substitute training or validation corpus. FineWeb remains the canonical data path for contest scoring.
The dataset captures three things:
V17 512Cube routing concepts translated into English.
Contest and submission guardrails for legal, reproducible BPB reduction.
Screenshot-derived scouting observations… See the full description on the dataset page: https://huggingface.co/datasets/8Planetterraforming/Parameter-Golf-V17-512Cube-XYZ-Priority-Orbital-Compression.vision-token-compression-bench
OPTIC-Bench
Optical Text In-Context Benchmark: how reliably do LLMs consume text
delivered as rendered images versus plain text tokens?
In summary, the evaluation reported here finds that optical text compression
is effective only within a narrow and specific envelope. Delivering content
as rendered images genuinely reduces input tokens, by thirteen to
fifty-four per cent depending on the model and the language, but only when
the document is long, the rendering is dense and the… See the full description on the dataset page: https://huggingface.co/datasets/translorentz/vision-token-compression-bench.all-deletion-compressionswikipedia-deletion-compressionssentence-compression
Dataset Card for Sentence Compression
This dataset is a collection of text-simplified pairs from the Sentence Compression project. See Sentence Compression for additional information.
This dataset can be used directly with Sentence Transformers to train embedding models.
Dataset Subsets
pair subset
Columns: "text", "simplified"
Column types: str, str
Examples:{
'text': "The USHL completed an expansion draft on Monday as 10 players who were on the rosters of… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/sentence-compression.sentence-compression
Dataset Card for "sentence-compression"
Dataset Summary
Dataset with pairs of equivalent sentences.
The dataset is provided "AS IS" without any warranty, express or implied.
Google disclaims all liability for any damages, direct or indirect, resulting from using the dataset.
Disclaimer: The team releasing sentence-compression did not upload the dataset to the Hub and did not write a dataset card. These steps were done by the Hugging Face team.
Supported Tasks… See the full description on the dataset page: https://huggingface.co/datasets/embedding-data/sentence-compression.compression-aware-abstention-data
Compression-Aware Abstention — labeled datasets
Compression-labeled training and evaluation data for "Compression-Aware
Abstention" (GroundLM @ EMNLP 2026). Derived from MuSiQue 2-hop QA
(CC BY 4.0, Trivedi et al., TACL 2022).
Each record pairs a MuSiQue example with a KV-cache compressor's per-token
survival mask at one of six retention ratios, plus the label that mask implies.
Fields
Field
Meaning
id
MuSiQue example id (2hop__*)
ratio
Target… See the full description on the dataset page: https://huggingface.co/datasets/moalikhoda/compression-aware-abstention-data.compression_datasetllm-compressionThis is the compression corpora dataset used in the paper "Compression Represents Intelligence Linearly".
We find that LLMs’ intelligence – reflected by benchmark scores – almost linearly correlates with their ability to compress external text corpora. We measure intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning, and provide the corresponding compression corpora here respectively named cc, python, and arxiv_math.
Load the data… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/llm-compression.fw-edu-cl100kcai-semantic-equivalence-benchmark
Contradish CAI-Bench
The semantic equivalence benchmark from Contradish
Do AI systems give the same answer when the wording changes but the meaning does not?
Contradish CAI-Bench measures semantic invariance: whether an AI system remains behaviorally consistent across prompts that express the same intent in different words.
This Hugging Face release contains 420 human-readable prompt pairs across 19 domains. Contradish is the official benchmark runner, scoring… See the full description on the dataset page: https://huggingface.co/datasets/compressionawareintelligence/cai-semantic-equivalence-benchmark.compression_test2compression_test3ImageNet-C-jpeg_compression-severity_4