datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
raw_v0.1_parquet
Common Pile v0.1 — Parquet Consolidated
Description
This dataset bundles all “raw” corpora from the Common Pile v0.1 Raw Data collection, converted to Apache Parquet and consolidated in a single repository.
Nothing has been filtered or modified; the only changes are:
Format: original JSON → Parquet
Layout: many repositories → one consolidated dataset
Extra column: a len_category bucket for quick length-based filtering
Only the three original columns (id, text… See the full description on the dataset page: https://huggingface.co/datasets/common-pile/raw_v0.1_parquet.AmericanStories-parquet
AmericanStories (parquet)
A parquet-native reformat of dell-research-harvard/AmericanStories — article-level full text of ~20 million U.S. newspaper scans (1774–1963) from the Library of Congress's Chronicling America collection, originally extracted by Dell et al. (arXiv:2308.12477).
This repo exists so the dataset loads in one line with the standard datasets / polars / pyarrow / dask stack, with no custom loading script and full Dataset Viewer support on the Hub.… See the full description on the dataset page: https://huggingface.co/datasets/biglam/AmericanStories-parquet.finqa-parquet
FinQA - Parquet
Modernized version of ibm-research/finqa in Parquet format.
Original dataset uses a deprecated loading script. This version is pre-processed and ready to use with modern datasets library.
Structure
Each row contains a single text field: pre_text + "\n" + question + "\n" + post_text
Rows: ~8,000 QA pairs (train + dev + test)
Source: ~2,800 financial reports
Split: all data combined (use your own split logic)
Usage
from datasets… See the full description on the dataset page: https://huggingface.co/datasets/whpthomas/finqa-parquet.blbooks-parquet
Dataset Card for British Library Books
This dataset is the same as https://huggingface.co/datasets/TheBritishLibrary/blbooks, however, this version is stored as parquet to avoid needing to run a datasets script. This also makes loading this dataset much quicker.
Dataset Summary
This dataset consists of books digitised by the British Library in partnership with Microsoft. The dataset includes ~25 million pages of out of copyright texts. The majority of the texts were… See the full description on the dataset page: https://huggingface.co/datasets/biglam/blbooks-parquet.deepplanning-parquet
DeepPlanning Parquet Standardized
This dataset is a Parquet-standardized release of DeepPlanning benchmark assets, built for analytics, reproducibility, and easier programmatic usage.
Source And Provenance
Raw dataset: Qwen/DeepPlanning
Benchmark code: QwenLM/Qwen-Agent/benchmark/deepplanning
Paper: DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints
Maintainer: tuandunghcmut
Important note:
The raw HF dataset contains environment… See the full description on the dataset page: https://huggingface.co/datasets/tuandunghcmut/deepplanning-parquet.tatoeba_mt_parquet
Dataset Card for DigitalLearningGmbH/tatoeba_mt_parquet
This is a mirror of Helsinki-NLP/tatoeba_mt, converted to parquet for compatibility with newer huggingface requirements.
Original dataset card follows.
Dataset Summary
The Tatoeba Translation Challenge is a multilingual data set of machine translation benchmarks derived from user-contributed translations collected by Tatoeba.org and provided as parallel corpus from OPUS. This dataset includes test and development… See the full description on the dataset page: https://huggingface.co/datasets/DigitalLearningGmbH/tatoeba_mt_parquet.dclm-baseline-1.0-parquet_urls
Dataset Card for dclm-baseline-1.0-parquet_urls
This dataset provides the URLs and top-level domains associated with training records in mlfoundations/dclm-baseline-1.0-parquet. It is part of a collection of datasets curated to make exploring LLM training datasets more straightforward and accessible.
Dataset Details
Dataset Description
This dataset was created by downloading the source data, extracting URLs and top-level domains, and retaining only… See the full description on the dataset page: https://huggingface.co/datasets/nhagar/dclm-baseline-1.0-parquet_urls.cuad-parquet
CUAD (Contract Understanding Attic Dataset) - Parquet
Modernized version of theatticusproject/cuad in Parquet format.
Original dataset uses a deprecated loading script. This version is pre-processed and ready to use with modern datasets library.
Structure
Each row contains a single text field: context + "\n" + question
Rows: 20,910 QA pairs
Source: 510 legal contracts
Split: train (all data, use your own split logic)
Usage
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/whpthomas/cuad-parquet.Parquet_FIles
Parquet_Files
Cross Language (CL) Datasets
Four datasets of language pair translations originating from CORDIS Project News (https://elrc-share.eu/)
Structured as follows:
| Field | Description |
| --------------- | ----------------------------------------------------------------------- |
| de/es/fr/it | Non-English transcripts of sentences |
| en… See the full description on the dataset page: https://huggingface.co/datasets/iix/Parquet_FIles.alfworld_parquet
ALFWorld parquet Dataset Card
由 make_dataset.py 从 ALFWorld json_2.1.1 原始游戏生成的parquet格式训练集/验证集，只含任务初始状态，大大减小数据空间。
1. 生成方式
对每个通过过滤的游戏 game.tw-pddl：在子进程中 reset 一次 TextWorld 环境
（max_episode_steps=50，与训练一致），提取 task / 初始 observation /
admissible actions，组装为初始 messages；环境随即关闭，仅留下
game_file 路径供训练期重建。
游戏发现规则：
<split_dir>/**/game.tw-pddl，路径含 movable 或 Sliced 的跳过；
同目录 traj_data.json 必须存在，且其 task_type 在六类白名单内；
game.tw-pddl 内 game_data["solvable"] is True。
2.… See the full description on the dataset page: https://huggingface.co/datasets/millyhua/alfworld_parquet.ID_Supreme_Court_Parquet
💎 Indonesian Supreme Court Parquet Dataset (ID_Supreme_Court_Parquet)
This repository provides a high-performance, compressed version of the Indonesian Supreme Court (Mahkamah Agung RI) court decisions. By converting raw legal data into the Apache Parquet format, this dataset is optimized for large-scale data engineering, fast I/O, and seamless integration with modern AI training pipelines. 🚀
💡 The Concept: Performance-First Legal Data
While HTML and JSON are great… See the full description on the dataset page: https://huggingface.co/datasets/Azzindani/ID_Supreme_Court_Parquet.tiny-pajama-parquet
TinyPajama
Subset of SlimPajama
Approximately 10B tokens
blbooks-parquet-embedded
Dataset Card for "blbooks-parquet-embedded"
More Information needed
sea-javanese-cleaned-parquet-v1
SEA Javanese Cleaned Parquet v1
Dataset Summary
This dataset is a cleaned Javanese pretraining corpus exported in Hugging Face parquet format.
Current public sources used in this release:
HuggingFaceFW/fineweb-2 / jav_Latn
allenai/c4 / jv
afrizalha/Centhini-1-Javanese
Cleaning and Deduplication
Current pipeline:
basic text cleaning
short-text filtering
repetition filtering
rule-based noise filtering
document-level exact deduplication across all included… See the full description on the dataset page: https://huggingface.co/datasets/rinnieyoung/sea-javanese-cleaned-parquet-v1.ft_dataset_parquet
89.7k筆 b8.3-patch3: b8.3-patch3.parquet
89.7k筆 b8.3-patch3: b8.3-patch3.parquet
89.7k筆 b8.3-patch3: b8.3-patch3.parquet
21.2k筆 scienceqa_zh: scienceqa_zh_train.parquet, scienceqa_zh_test.parquet, scienceqa_zh_validation.parquet
b8.3-patch3
Based on b8.3-patch2, replace two translation dataset.
Remove ultrachat-MixtralTranslation and ultrachat-GoogleTranslation
Add GPT4o-Translation-zh2en and GPT4o-Translation-en2zh.
Source
Numbers
GPT4o-Translation-zh2en
12,408… See the full description on the dataset page: https://huggingface.co/datasets/c00cjz00/ft_dataset_parquet.