datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
llm-compressionThis is the compression corpora dataset used in the paper "Compression Represents Intelligence Linearly".
We find that LLMs’ intelligence – reflected by benchmark scores – almost linearly correlates with their ability to compress external text corpora. We measure intelligence along three key abilities: knowledge and commonsense, coding, and mathematical reasoning, and provide the corresponding compression corpora here respectively named cc, python, and arxiv_math.
Load the data… See the full description on the dataset page: https://huggingface.co/datasets/hkust-nlp/llm-compression.semantic-compression-sft
sematic-compression-sft
Dataset Summary
sematic-compression-sft is a synthetic supervised fine-tuning dataset for semantic compression.
The task is to convert verbose natural-language or code inputs into compact outputs that preserve reasoning-relevant information.
This dataset is designed for training compression models/adapters used before downstream LLM inference to reduce prompt size while retaining functional utility.
Goal
The objective is not generic… See the full description on the dataset page: https://huggingface.co/datasets/Sudhendra/semantic-compression-sft.Fathom-V0.4-RL-Compression