datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
dclm-baseline-1.0-parquet
DCLM-baseline
Note: this is an identical copy of https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0, where all the files have been mapped to a parquet format.
DCLM-baseline is a 4T token / 3B document pretraining dataset that achieves strong performance on language model benchmarks.
Below are comparisions of model trained on DCLM-baseline with other models in the 7B regime.
Model
Params
Tokens
Open dataset?
CORE
MMLU
EXTENDED
Open weights, closed datasets… See the full description on the dataset page: https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0-parquet.ettin-parquetTCGA-12K-parquet
TCGA-12K Parquet
Attribution
This dataset contains 224 x 224 JPEG patches from whole-slide images originally downloaded from The Cancer Genome Atlas (TCGA) that are available in the NCI Genomic Data Commons (GDC) Open Access tier. We mirror and repackage a commonly used ~12k WSI subset in parquet format for ease of training. We exclude patches that did not pass HSV thresholding, following the procedure in Kaiko.AI's Midnight paper. Patches were randomly sampled across… See the full description on the dataset page: https://huggingface.co/datasets/medarc/TCGA-12K-parquet.nfpc-parquet-dataset
AML Mule Account Detection Challenge
Data Format: All files are in Apache Parquet format (Snappy compression). Use pandas.read_parquet(), pyarrow.parquet, or any Parquet-compatible reader. Transaction files are split across batch-N/ subdirectories.
Primary Objective/Problem Statement
Identify mule accounts used for money laundering from banking transaction and account data. Given labelled training data and unlabelled test accounts, predict which test accounts are mules.… See the full description on the dataset page: https://huggingface.co/datasets/preetisheoran/nfpc-parquet-dataset.hypersim-episodes-v3-parquet
hypersim-episodes-v3-parquet
Per-frame Parquet dataset for ReCAST tracker training.
Schema
One row per frame, grouped by episode_id. Arrow memory-mapped access
enables reading specific frames without loading entire episodes.
Column
Type
Description
episode_id
int32
Episode identifier
frame_idx
int32
Frame index within episode
jpeg
binary
JPEG-encoded RGB frame
depth
list<float32>
Flat H×W depth map
seg
list<uint16>
Semantic segmentation (empty if… See the full description on the dataset page: https://huggingface.co/datasets/OSResight/hypersim-episodes-v3-parquet.tartanair-episodes-v3-parquet
tartanair-episodes-v3-parquet
Per-frame Parquet dataset for ReCAST tracker training.
Schema
One row per frame, grouped by episode_id. Arrow memory-mapped access
enables reading specific frames without loading entire episodes.
Column
Type
Description
episode_id
int32
Episode identifier
frame_idx
int32
Frame index within episode
jpeg
binary
JPEG-encoded RGB frame
depth
list<float32>
Flat H×W depth map
seg
list<uint16>
Semantic segmentation (empty if… See the full description on the dataset page: https://huggingface.co/datasets/OSResight/tartanair-episodes-v3-parquet.NQ-F_1min_OHLCV_Parquetusc-x-24-us-election-parquetThis is a version of the USC X 24 US Election Twitter/X Dataset from USC, cleaned and converted to parquet.
The repository contains multiple directories named part_{part_number}, where each directory consists of chunk files prefixed with a timeline. Each chunk file contains 50,000 tweets related to the US elections 2024. Specifically, each subdirectory labeled with the prefix "part" contains 20 chunk files, resulting in a total of 1,000,000 tweets per part... Check out our memo that provides… See the full description on the dataset page: https://huggingface.co/datasets/deadbirds/usc-x-24-us-election-parquet.blbooks-parquet
Dataset Card for British Library Books
This dataset is the same as https://huggingface.co/datasets/TheBritishLibrary/blbooks, however, this version is stored as parquet to avoid needing to run a datasets script. This also makes loading this dataset much quicker.
Dataset Summary
This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were… See the full description on the dataset page: https://huggingface.co/datasets/biglam/blbooks-parquet.full-fold-the-rag-parquet-merged0222TCGA-12K-parquet-shuffled
TCGA-12K Parquet (Shuffled)
Attribution
This dataset contains 224 x 224 JPEG patches from whole-slide images originally downloaded from The Cancer Genome Atlas (TCGA) that are available in the NCI Genomic Data Commons (GDC) Open Access tier. We mirror and repackage a commonly used ~12k WSI subset in parquet format for ease of training. We exclude patches that did not pass HSV thresholding, following the procedure in Kaiko.AI's Midnight paper. Patches were randomly sampled… See the full description on the dataset page: https://huggingface.co/datasets/medarc/TCGA-12K-parquet-shuffled.CICIOT2023-PARQUET
CICIoT2023 — ipfixprobe flow records (Parquet)
1,479,074,715 bidirectional network flows re-exported from the raw PCAPs of
CICIoT2023 (Canadian
Institute for Cybersecurity, University of New Brunswick) with
ipfixprobe 5.7.0, stored as 308 Parquet
files (~19.4 GB) covering 33 attack classes + benign traffic from the
105-device IoT testbed.
The original dataset ships ~587 GB of PCAPs and CSV features computed with a
closed pipeline. This conversion provides an alternative… See the full description on the dataset page: https://huggingface.co/datasets/Lystea/CICIOT2023-PARQUET.IndicVoice-latent-NEW-parquetDanbooru-2026-parquet-metadatapickup-carrot-remove-parquet-metadata-2This dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"trossen_subversion": "v1.0",
"robot_type": "trossen_ai_stationary",
"total_episodes": 21,
"total_frames": 9383,
"total_tasks": 1,
"total_videos": 84,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:21"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet"… See the full description on the dataset page: https://huggingface.co/datasets/argus-systems/pickup-carrot-remove-parquet-metadata-2.institutional_investors_parquet_by_stockhover-parquet
Dataset Card for HoVer (Parquet Format)
Note: This is a scriptless, Parquet-based version of the HoVer dataset for seamless integration with HuggingFace datasets library. No trust_remote_code required!
Quick Start
from datasets import load_dataset
# Load the dataset (no trust_remote_code needed!)
dataset = load_dataset("vincentkoc/hover-parquet")
# Access splits
train = dataset["train"]
validation = dataset["validation"]
test = dataset["test"]
# Example usage… See the full description on the dataset page: https://huggingface.co/datasets/vincentkoc/hover-parquet.StockChina-Minute-Parquet
China Stock Market 1-Minute Bar Dataset (Parquet)
This dataset provides high-frequency 1-minute historical candlestick and trading data for Chinese A-Share stocks (SSE / SZSE: .XSHE, .XSHG).
Attribution & Source Credit
This dataset is an optimized Parquet conversion of the original CSV dataset created by jobs-git:
Original Dataset: jobs-git/StockChina-Minute
Original uncompressed CSV size: ~121.5 GB across 1,450 stock symbols.
Optimizations in this… See the full description on the dataset page: https://huggingface.co/datasets/q1232990/StockChina-Minute-Parquet.IOT23-PARQUET
IoT-23 — canonical flow parquet (light path)
IoT-23 (Stratosphere
Laboratory, CTU University: real IoT malware infections + benign IoT device
captures) converted from the light distribution's labeled Zeek conn logs
into a canonical flow-record parquet schema: 23 captures, 325.3M rows,
7.8 GB zstd. One parquet per capture — leave-one-capture-out splits rebuild
from filenames.
Fidelity caveats, by construction (Tier A only):
Source is conn.log.labeled, not pcap: TCP flag… See the full description on the dataset page: https://huggingface.co/datasets/Lystea/IOT23-PARQUET.Danbooru-2026-parquet-metadataorca_parquet
ORCA – Arabic Language Understanding Benchmark
Converted from the original wissamantoun/orca_hf dataset script to parquet files.
Usage
from datasets import load_dataset
# Load a specific config
ds = load_dataset("PATH/orca_parquet", name="sentiment")
# Load all configs
ds = load_dataset("PATH/orca_parquet")
nucl-parquet-data
Licensing. These shards are derived from ENDF/B-VIII.0, a US Government
work — public domain in the US under 17 U.S.C. §105 — in the NJOY-processed
pointwise form published by the OpenMC project. Neither the evaluation nor its
processed form is ours to relicense, so the previous license: mit tag on this
dataset was incorrect and has been removed. MIT covers the nucl-parquet code
and conversion, not the bundled evaluated data. Per-library terms are recorded
in data/licenses.toml.
Cite: D.A.… See the full description on the dataset page: https://huggingface.co/datasets/gerchowl/nucl-parquet-data.ei-sand-challenge-2026-parquetparquet_for_narprojpickup-carrot-remove-parquet-metadataThis dataset was created using LeRobot.
Dataset Structure
meta/info.json:
{
"codebase_version": "v2.1",
"trossen_subversion": "v1.0",
"robot_type": "trossen_ai_stationary",
"total_episodes": 21,
"total_frames": 9383,
"total_tasks": 1,
"total_videos": 84,
"total_chunks": 1,
"chunks_size": 1000,
"fps": 30,
"splits": {
"train": "0:21"
},
"data_path": "data/chunk-{episode_chunk:03d}/episode_{episode_index:06d}.parquet"… See the full description on the dataset page: https://huggingface.co/datasets/argus-systems/pickup-carrot-remove-parquet-metadata.flores-parquet
FLORES Parquet
Parquet version of the FLORES dataset for efficient streaming.
⚠️ This is a derivative work: This dataset is a reformatted version of the original FLORES dataset created by Meta AI. All credit goes to the original authors. This version simply converts the data to Parquet format for easier streaming and usage.
Usage
from datasets import load_dataset
# Load specific language
ds = load_dataset("tomasmajercik/flores-parquet", name="fra_Latn"… See the full description on the dataset page: https://huggingface.co/datasets/tomasmajercik/flores-parquet.fuss-parquet
FUSS Parquet Dataset
This dataset provides the Free Universal Sound Separation (FUSS) Dataset as a set of parquet files.
The Free Universal Sound Separation (FUSS) Dataset is a database of arbitrary sound mixtures and source-level references, for use in experiments on arbitrary sound separation.
This is the official sound separation data for the DCASE2020 Challenge Task 4: Sound Event Detection and Separation in Domestic Environments.
Overview: FUSS audio data is sourced from a… See the full description on the dataset page: https://huggingface.co/datasets/scaleinvariant/fuss-parquet.synthia-rand-cityscapes-16class-parquet_fixdsb_parquetcongressional-record-parquet
Congressional Record 43rd–114th Congresses — Parquet Edition
This repository contains a Parquet-formatted derivative of the Stanford
Congressional Record for the 43rd–114th Congresses: Parsed Speeches and Phrase Counts
dataset.
The files were prepared as a compact, query-friendly research corpus for historical
text search with tools such as DuckDB and the Congressional Record Explorer.
Coverage
Congresses: 43rd–114th
One Parquet file per Congress
Files:… See the full description on the dataset page: https://huggingface.co/datasets/yeeder/congressional-record-parquet.