datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
Language-Grounded_Sparse_Encoder_Training
Language-Grounded Sparse Encoder (LanSE) — Training Data
This repository hosts the AI-generated images and human annotation datasets accompanying the paper:
Human-like Content Analysis for Generative AI with Language-Grounded Sparse Encoders
Yiming Tang, Arash Lagzian, Srinivas Anumasa, Qiran Zou, Yingtao Zhu, Ye Zhang, Trang Nguyen, Yih-Chung Tham, Ehsan Adeli, Ching-Yu Cheng, Yilun Du, Dianbo Liu
National University of Singapore · Tsinghua University · Stanford University ·… See the full description on the dataset page: https://huggingface.co/datasets/DesmondYMTang2024/Language-Grounded_Sparse_Encoder_Training.SparseVideoNav
SparseVideoNav Datasets
This repository contains the real-world navigation datasets released with OpenDriveLab/SparseVideoNav:
BVN: Beyond-the-View Navigation.
IFN: Instruction-Following Navigation.
Project links:
Project page: https://opendrivelab.com/SparseVideoNav
GitHub: https://github.com/OpenDriveLab/SparseVideoNav
Paper: https://arxiv.org/abs/2602.05827
Dataset Summary
SparseVideoNav studies real-world vision-language navigation with sparse future… See the full description on the dataset page: https://huggingface.co/datasets/OpenDriveLab/SparseVideoNav.sparse-reward-long-tasks
Sparse Reward Long Tasks
Rights & intended use: legacy public research corpus / portfolio
artifact. Hosted frontier-model outputs are research-only inputs under
project policy (synthetic-factory#161):
intended_use: research_only, project_training_policy: blocked. Not
training data for any model-weight update. Machine-readable record:
rights.json.
Release status: The raw, uncurated payload is now published under
data/raw/. It is available for inspection and reproducibility… See the full description on the dataset page: https://huggingface.co/datasets/rmems/sparse-reward-long-tasks.sparse-probing
Sparse Probing Datasets
155 binary classification tasks for probing language model representations.
From: "Are Sparse Autoencoders Useful? A Case Study in Sparse Probing" (arXiv:2502.16681)
Source: EleutherAI/sae-probes
Usage
from datasets import load_dataset
# Load a specific dataset
ds = load_dataset("serteal/sparse-probing", "87_glue_cola")
# List available configurations
from datasets import get_dataset_config_names
configs =… See the full description on the dataset page: https://huggingface.co/datasets/serteal/sparse-probing.Dr.Sparse-OTF-test-set
Dr.Sparse OTF Test Set
100 sparse matrices from the SuiteSparse Matrix Collection,
converted to the flat binary format the Dr.Sparse
benchmark harness reads. This is the held-out evaluation set for LLM-generated
CUDA sparse kernels (SpMV / SpMM / SpGEMM), kept separate from the matrices the
models were developed against.
Layout
Matrices are grouped into size tiers by row count, the convention Dr.Sparse task
discovery scans for:
tier
rows
matrices
size… See the full description on the dataset page: https://huggingface.co/datasets/KinGeorge/Dr.Sparse-OTF-test-set.SparseTable_Bench_Datasetmultinews_sparse_oracleThis is a copy of the Multi-News dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The summary field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "oracle", i.e. the number of documents retrieved, k, is set as the original number of input documents for each example
Retrieval results on the… See the full description on the dataset page: https://huggingface.co/datasets/allenai/multinews_sparse_oracle.walnut-edip-sparse20
Scitomo Walnut EDIP Sparse-20 prepared dataset
This is a derived Scitomo Sparse-20 preparation of the public Walnut-1
cone-beam X-ray CT acquisition. It is not the original Walnut archive, not a
published EDIP reconstruction, and not a blessed Scitomo result. The package
contains measured projections, corrected vector-cone geometry, and the
published AGD-50 evaluation reference used by the maintained Scitomo Walnut
EDIP evidence workflow.
Provenance and attribution… See the full description on the dataset page: https://huggingface.co/datasets/scitomo/walnut-edip-sparse20.multinews_sparse_maxThis is a copy of the Multi-News dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The summary field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "max", i.e. the number of documents retrieved, k, is set as the maximum number of documents seen across examples in this dataset, in this case… See the full description on the dataset page: https://huggingface.co/datasets/allenai/multinews_sparse_max.ms2_sparse_maxThis is a copy of the MS^2 dataset, except the input source documents of its validation split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The background field of each example
corpus: The union of all documents in the train, validation and test splits. A document is the concatenation of the title and abstract.
retriever: BM25 via PyTerrier with default settings
top-k strategy: "max", i.e. the number of documents retrieved, k, is set as the maximum number of… See the full description on the dataset page: https://huggingface.co/datasets/allenai/ms2_sparse_max.multinews_sparse_meanThis is a copy of the Multi-News dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The summary field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "mean", i.e. the number of documents retrieved, k, is set as the mean number of documents seen across examples in this dataset, in this case k==3… See the full description on the dataset page: https://huggingface.co/datasets/allenai/multinews_sparse_mean.simplemath-ita-sparse
Dataset Card for "simplemath-ita-sparse"
More Information needed
multilingual-automotive-sparse
Multilingual Automotive Sparse Retrieval Corpus
A synthetic training corpus for fine-tuning multilingual sparse retrieval
models (SPLADE-family) on automotive, supply-chain, and geopolitics content.
Designed to teach a model cross-lingual alignment: a Japanese query should
retrieve the right English passage, and vice versa.
Corpus contents
File
Rows
Purpose
concepts.jsonl
4995
Raw concept records
train_triplets.jsonl
134880
Flattened (query, positive… See the full description on the dataset page: https://huggingface.co/datasets/cp500/multilingual-automotive-sparse.ms2_sparse_meanThis is a copy of the MS^2 dataset, except the input source documents of its validation split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The background field of each example
corpus: The union of all documents in the train, validation and test splits. A document is the concatenation of the title and abstract.
retriever: BM25 via PyTerrier with default settings
top-k strategy: "mean", i.e. the number of documents retrieved, k, is set as the mean number of… See the full description on the dataset page: https://huggingface.co/datasets/allenai/ms2_sparse_mean.Dr.Sparse-RL-level4
Dr.Sparse RL rollout pool, level4_huge members
The SpGEMM online-RL rollout pool is 130 SuiteSparse matrices: the 118 level1-3
matrices in the Dr.Sparse git repo (level_sparse/level{1_small,2_medium,3_large})
plus the 12 level4 matrices in this dataset. It is disjoint from the 100-matrix
OTF test set.
The 12 level4 (>= 1.1M rows) matrices, in
Dr.Sparse, in the repo's raw .bin
layout ([int32 rows, cols, nnz][int32 row_ptr][int32 col_ind][float32 values][float32 x],
read by… See the full description on the dataset page: https://huggingface.co/datasets/KinGeorge/Dr.Sparse-RL-level4.cochrane_sparse_maxThis is a copy of the Cochrane dataset, except the input source documents of its validation split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The target field of each example
corpus: The union of all documents in the train, validation and test splits. A document is the concatenation of the title and abstract.
retriever: BM25 via PyTerrier with default settings
top-k strategy: "max", i.e. the number of documents retrieved, k, is set as the maximum number of… See the full description on the dataset page: https://huggingface.co/datasets/allenai/cochrane_sparse_max.wcep_sparse_oracleThis is a copy of the WCEP-10 dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The summary field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "oracle", i.e. the number of documents retrieved, k, is set as the original number of input documents for each example
Retrieval results on the train… See the full description on the dataset page: https://huggingface.co/datasets/allenai/wcep_sparse_oracle.multixscience_sparse_oracleThis is a copy of the Multi-XScience dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The related_work field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "oracle", i.e. the number of documents retrieved, k, is set as the original number of input documents for each example
Retrieval results… See the full description on the dataset page: https://huggingface.co/datasets/allenai/multixscience_sparse_oracle.Dr.Sparse-RL-train-562
Dr.Sparse SpGEMM training pool (562 matrices)
The complete RL / selector training pool of Dr.Sparse (branch v2): 562 SuiteSparse matrices in the
harness .bin layout (int32 rows, cols, nnz; int32 row_ptr; int32 col_ind; float32 values; float32 x),
laid out as level1_small/ (91), level2_medium/ (273), level3_large/ (198); the huge tier is deliberately left out of training and evaluation.
Every matrix has a cuSPARSE SpGEMM reference (C = AA, or AA^T when rectangular) on an H200;… See the full description on the dataset page: https://huggingface.co/datasets/KinGeorge/Dr.Sparse-RL-train-562.stack-v2-sparse-classes-10k
Stack v2 Sparse Python Classes 10k
This is a 10,000-sample snapshot for Diffusion + Autoregressive hybrid code generation experiments.
Source
The data is extracted from bigcode/the-stack-v2-dedup, Python subset. The extraction uses Stack v2 metadata as source of truth, groups candidates by repo_name + revision_id, fetches files with git partial fetch + sparse checkout, then applies AST-level class filters.
Splits
train.jsonl: 9,000
val.jsonl: 500
test.jsonl:… See the full description on the dataset page: https://huggingface.co/datasets/hybrid-diff-ar/stack-v2-sparse-classes-10k.multixscience_sparse_maxThis is a copy of the Multi-XScience dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The related_work field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "max", i.e. the number of documents retrieved, k, is set as the maximum number of documents seen across examples in this dataset, in this… See the full description on the dataset page: https://huggingface.co/datasets/allenai/multixscience_sparse_max.ms2_sparse_oracleThis is a copy of the MS^2 dataset, except the input source documents of its validation split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The background field of each example
corpus: The union of all documents in the train, validation and test splits. A document is the concatenation of the title and abstract.
retriever: BM25 via PyTerrier with default settings
top-k strategy: "oracle", i.e. the number of documents retrieved, k, is set as the original number… See the full description on the dataset page: https://huggingface.co/datasets/allenai/ms2_sparse_oracle.wcep_sparse_maxThis is a copy of the WCEP-10 dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The summary field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "max", i.e. the number of documents retrieved, k, is set as the maximum number of documents seen across examples in this dataset, in this case k==10… See the full description on the dataset page: https://huggingface.co/datasets/allenai/wcep_sparse_max.long_sparse_unstructured_tableThis dataset is generated syhthetically to create tables with following characteristics:
Empty cell percentage in following range [40,70] (Sparse)
There is no seperator between rows and columns (un-structured).
15 <= num rows <= 30, 7 <= num columns <= 15 (Long)
Load the dataset
import io
import pandas as pd
from PIL import Image
def bytes_to_image(self, image_bytes: bytes):
return Image.open(io.BytesIO(image_bytes))
def parse_annotations(self, annotations: str) ->… See the full description on the dataset page: https://huggingface.co/datasets/nanonets/long_sparse_unstructured_table.multixscience_sparse_meanThis is a copy of the Multi-XScience dataset, except the input source documents of its test split have been replaced by a sparse retriever. The retrieval pipeline used:
query: The related_work field of each example
corpus: The union of all documents in the train, validation and test splits
retriever: BM25 via PyTerrier with default settings
top-k strategy: "mean", i.e. the number of documents retrieved, k, is set as the mean number of documents seen across examples in this dataset, in this… See the full description on the dataset page: https://huggingface.co/datasets/allenai/multixscience_sparse_mean.small_sparse_structured_tableThis dataset is generated syhthetically to create tables with following characteristics:
Empty cell percentage in following range [40,70] (Sparse)
There is clear seperator between rows and columns (Structured).
4 <= num rows <= 10, 2 <= num columns <= 6 (Small)
Load the dataset
import io
import pandas as pd
from PIL import Image
def bytes_to_image(self, image_bytes: bytes):
return Image.open(io.BytesIO(image_bytes))
def parse_annotations(self, annotations: str) ->… See the full description on the dataset page: https://huggingface.co/datasets/nanonets/small_sparse_structured_table.stack-v2-sparse-classes-36k
Stack v2 Sparse Python Classes 36k
This is a 36,000-sample snapshot for Diffusion + Autoregressive hybrid code generation experiments.
Source
The data is extracted from bigcode/the-stack-v2-dedup, Python subset. The extraction uses Stack v2 metadata as source of truth, groups candidates by repo_name + revision_id, fetches files with git partial fetch + sparse checkout, then applies AST-level class filters.
Splits
train.jsonl: 35,000
val.jsonl: 500
test.jsonl:… See the full description on the dataset page: https://huggingface.co/datasets/hybrid-diff-ar/stack-v2-sparse-classes-36k.stack-v2-sparse-classes-75kplus
Stack v2 Sparse Python Classes 75kplus
This is a frozen snapshot with 75829 samples for Diffusion + Autoregressive hybrid code generation experiments.
Splits
train.jsonl: 74829
val.jsonl: 500
test.jsonl: 500
all.jsonl: 75829
Source
The data is extracted from bigcode/the-stack-v2-dedup, Python subset. The extraction uses Stack v2 metadata as source of truth, groups candidates by repo_name + revision_id, fetches files with git partial fetch + sparse checkout… See the full description on the dataset page: https://huggingface.co/datasets/hybrid-diff-ar/stack-v2-sparse-classes-75kplus.HymathRec_6K_sparsesmall_sparse_unstructured_tableThis dataset is generated syhthetically to create tables with following characteristics:
Empty cell percentage in following range [40,70] (Sparse)
There is no seperator between rows and columns (un-structured).
4 <= num rows <= 10, 2 <= num columns <= 6 (Small)
Load the dataset
import io
import pandas as pd
from PIL import Image
def bytes_to_image(self, image_bytes: bytes):
return Image.open(io.BytesIO(image_bytes))
def parse_annotations(self, annotations: str) ->… See the full description on the dataset page: https://huggingface.co/datasets/nanonets/small_sparse_unstructured_table.