datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
SCBench-preprocessedThis is the preprocessed version of Microsoft SCBench, used by KVzip:
Each data example has a format of {context: str, question: List[str], answers: List[str]}
Each dataset contains only examples whose context token length (measured with the LLaMA3 tokenizer) is less than 125K, fitting within the context limit of LLaMA3 models.
We also provide shortened versions of SCBench, excluding tasks {choice_eng, qa_eng, and vt}, which are difficult to shorten.
The "tiny" tag (e.g., scbench_kv_tiny)… See the full description on the dataset page: https://huggingface.co/datasets/Jang-Hyun/SCBench-preprocessed.SCBench
SCBench
[Paper]
[Code]
[Project Page]
SCBench (SharedContextBench) is a comprehensive benchmark to evaluate efficient long-context methods in a KV cache-centric perspective, analyzing their performance across the full KV cache lifecycle (generation, compression, retrieval, and loading) in real-world scenarios where context memory (KV cache) is shared and reused across multiple requests.
🎯 Quick Start
Load Data
You can download and load the SCBench data… See the full description on the dataset page: https://huggingface.co/datasets/microsoft/SCBench.SCBench
SCBench
[Paper]
[Code]
SCBench (SharedContextBench) is a comprehensive benchmark to evaluate efficient long-context methods in a KV cache-centric perspective, analyzing their performance across the full KV cache lifecycle (generation, compression, retrieval, and loading) in real-world scenarios where context memory (KV cache) is shared and reused across multiple requests.
Dataset
SCBench covers 12 diverse tasks that test four key long-context capabilities: string… See the full description on the dataset page: https://huggingface.co/datasets/MInference/SCBench.scbe-spine-overlay-proof-v1
Status: experimental. Experiment-specific slice. Primary public dataset: scbe-aethermoore-training-data.
SCBE Spine-Overlay Proof v1
Tiny demonstration bundle (18 rows = 3 domains x 6 tongues)
proving that the SCBE 12+ lane code-packet spine handles code,
chemistry, and mechanical motion as overlays on a single tokenizer
substrate, without forking the system.
Why this exists
Every row carries the same baseline lanes (binary, tokenizer, transport,
labels… See the full description on the dataset page: https://huggingface.co/datasets/issdandavis/scbe-spine-overlay-proof-v1.scbe-governance-receipts-v1
Status: experimental. Experiment-specific slice. Primary public dataset: scbe-aethermoore-training-data.
SCBE Governance Receipts v1
Schema: scbe_governed_dataset_v1
Receipt schema: scbe_governance_receipt_v1
Built: 2026-05-04T23:45:37Z
Rows: 40
Dataset ID: scbe-governance-receipts-v1
What this is
A governed dataset where every row carries a full 34-field SCBE
governance receipt (poly-embedded JEPA fingerprint + tri-vector
cross-braid hash + Sacred Egg ring seal… See the full description on the dataset page: https://huggingface.co/datasets/issdandavis/scbe-governance-receipts-v1.scbe-aethermoore-datasets
Status: experimental. Experiment-specific slice. Primary public dataset: scbe-aethermoore-training-data.
issdandavis/scbe-aethermoore-knowledge-base
Programmatic SCBE training package built from the local ledgered corpus.
Generated at: 2026-04-04T16:17:19.436108+00:00
Source file: training/ledgered/sft_ledgered_clean.jsonl
Audit status: ALLOW
Rows total: 15206
Train rows: 13685
Validation rows: 760
Test rows: 761
Positive pairs: 15206
Files
data/all.jsonl —… See the full description on the dataset page: https://huggingface.co/datasets/issdandavis/scbe-aethermoore-datasets.scbe-codeflow-bijective-v1
Status: experimental. Experiment-specific slice. Primary public dataset: scbe-aethermoore-training-data.
SCBE Codeflow Bijective v1
Supervised fine-tuning corpus teaching bijective multi-tongue / multi-language
code editing. Each algorithm is decomposed into N semantic slots. Every slot
is filled in all 6 Sacred Tongues. An edit at slot k in any tongue maps
deterministically to the parallel slot k in every other tongue. Syntactic line
counts may differ per tongue; semantic… See the full description on the dataset page: https://huggingface.co/datasets/issdandavis/scbe-codeflow-bijective-v1.scbe-life-science-research-training-demo
Status: experimental. Experiment-specific slice. Primary public dataset: scbe-aethermoore-training-data.
SCBE Research Training Package
This package was generated from live pubmed pulls for the query protein structure prediction and is meant for
lightweight Hugging Face dataset and SFT experiments.
Files
papers.jsonl: normalized raw research records
sft_train.jsonl: train split for instruction-style tasks
sft_validation.jsonl: validation split… See the full description on the dataset page: https://huggingface.co/datasets/issdandavis/scbe-life-science-research-training-demo.scbe-tongue-drill-sft-v1
Status: experimental. Experiment-specific slice. Primary public dataset: scbe-aethermoore-training-data.
SCBE Tongue Drill SFT v1
Supervised fine-tuning drill dataset for the SCBE Sacred Tongues table-lock system.
Each row is a 3-turn chat (system / user / assistant) teaching the model to emit
canonical packets verbatim for a given (map, tongue, value) triple.
Splits
Split
Rows
all
2630
train
2373
holdout
257
Holdout is row_index % 10 == 0… See the full description on the dataset page: https://huggingface.co/datasets/issdandavis/scbe-tongue-drill-sft-v1.scbench-under-130ksc_benchmarkscbe-drill-langues-full
Status: experimental. Research artifact, not a production candidate. Canonical dataset: scbe-aethermoore-training-data.
SCBE langues drill (full)
One file: drill_langues_full.jsonl.
Drill set over the six Sacred Tongues (KO, AV, RU, CA, UM, DR) used for
tokenizer and translation practice rows. Single split, no train/eval division -
if you need a held-out set, carve one yourself and group by semantic root so the
same item does not appear on both sides.
CommentarySet_full_data
CommentarySet
We have provided the code Code, and the complete CommentarySet data.
Data Structure:
For training and validating, you should arrange the dataset and code in the following structure:
YOUR_MODEL_ROOT_DIRECTORY
data
commentary
atheletics_final.json
basketball_final.json
...
video
athletics
001
5.mp4
7.mp4
...
...
basketball
...
train.json
test.json
Video LLMs Official Code(e.g VILA)
metric_six_dimensional.py
metric_traditional_gpt.py… See the full description on the dataset page: https://huggingface.co/datasets/SCBench/CommentarySet_full_data.