common-pile/raw_v0.1_parquet

Common Pile v0.1 — Parquet Consolidated

Description

This dataset bundles all “raw” corpora from the Common Pile v0.1 Raw Data [collection](https://huggingface.co/collections/common-pile/common-pile-v01-raw-data-6826b454a5a6a445d0b51b37), converted to Apache Parquet and consolidated in a single repository.

Nothing has been filtered or modified; the only changes are:

• —Format: original JSON → Parquet
• —Layout: many repositories → one consolidated dataset
• —Extra column: a len_category bucket for quick length-based filtering

Only the three original columns (id, text, source) are carried over; len_category is derived from text length.

Dataset Schema

License Issues

Licensing follows the individual corpora.

While we aim to produce datasets with completely accurate licensing information, license laundering and inaccurate metadata can cause us to erroneously assign the incorrect license to some documents (for further discussion of this limitation, please see our paper). If you believe you have found an instance of incorrect licensing in this dataset, please start a discussion on this repository.

Other Versions

• —Per-corpus “raw” datasets
• —Filtered set used for Comma v0.1 training

Citation

If you use this dataset, please cite:

@article{kandpal2025common,
  title       = {{The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text}},
  author      = {Nikhil Kandpal and Brian Lester and Colin Raffel and Sebastian Majstorovic and
                 Stella Biderman and Baber Abbasi and Luca Soldaini and Enrico Shippole and
                 A. Feder Cooper and Aviya Skowron and Shayne Longpre and Lintang Sutawika and
                 Alon Albalak and Zhenlin Xu and Guilherme Penedo and Loubna Ben and Elie Bakouch and
                 John David and Honglu Fan and Dashiell Stander and Guangyu Song and Aaron Gokaslan and
                 John Kirchenbauer and Tom Goldstein and Brian R and Bhavya Kailkhura and Tyler Murray},
  journal     = {arXiv preprint},
  year        = {2025}
}