CoolFace
Datasetpublic

anhchanghoangsg/reddit_pushshift_dataset_cleaned

🚀 Cleaned Reddit Pushshift Dataset (Parquet) 📖 Dataset Description This dataset is a highly optimized, meticulously cleaned, and structured version of the raw Reddit Pushshift dump. It has been transformed from massive .zst compressed JSON files into highly efficient, columnar Parquet files, making it immediately ready for Big Data analytics, SQL querying (via DuckDB), and Large Language Model (LLM) training. The dataset includes both Reddit Submissions (Posts)… See the full description on the dataset page: https://huggingface.co/datasets/anhchanghoangsg/reddit_pushshift_dataset_cleaned.

sourceHugging Faceupdated 6mo agoView on Hugging Face
3likes590downloads
Dataset Card

🚀 Cleaned Reddit Pushshift Dataset (Parquet)

📖 Dataset Description

This dataset is a highly optimized, meticulously cleaned, and structured version of the raw Reddit Pushshift dump. It has been transformed from massive .zst compressed JSON files into highly efficient, columnar Parquet files, making it immediately ready for Big Data analytics, SQL querying (via DuckDB), and Large Language Model (LLM) training.

The dataset includes both Reddit Submissions (Posts) and Comments.

🧹 Data Cleaning & Preprocessing Pipeline

Unlike the raw Pushshift dump which contains heavy noise, this dataset has been processed through a strict ETL pipeline:

  1. 1.Dead Data Removal: - Completely dropped rows where the author, selftext, or body is [deleted], [removed], or None.
  2. 2.Dropped comments missing essential relation links (link_id, parent_id).
  3. 3.Noise & Encoding Fixes:
  4. 4.Filtered out submissions containing severe Unicode/font errors (e.g., \ufffd characters).
  5. 5.Data Type Normalization:
  6. 6.Converted raw Unix timestamps (created_utc) into readable YYYY-MM-DD string format.
  7. 7.Safely cast numerical fields (score, num_comments, upvote_ratio, etc.) to strict Integer/Float types to prevent schema crashing.
  8. 8.Handled missing text fields by filling them with "None" to ensure consistent string types.
  9. 9.NSFW & Sensitive Content Filtering:
  10. 10.Applied a smart Regex and CamelCase keyword filter to automatically identify and exclude highly sensitive or purely NSFW subreddits from the repository.
  11. 11.Storage Optimization (The "Small File" Fix):
  12. 12.Micro-files (under 1MB) representing inactive subreddits were skipped. This prevents the "Small File Problem" in Parquet storage, ensuring the Hugging Face Data Viewer and distributed computing frameworks (like Apache Spark) run at lightning speed.

🗂️ Dataset Structure (Schema)

📝 Submissions (Posts)

FieldTypeDescription
authorstringThe Reddit username of the poster.
namestringUnique base36 ID of the submission (e.g., t3_xxx).
titlestringThe title of the post.
selftextstringThe text body of the post.
created_utcstringDate of creation (YYYY-MM-DD).
scoreint64Net upvotes minus downvotes.
upvote_ratiofloat64Ratio of upvotes to total votes.
num_commentsint64Number of comments on the thread.
num_crosspostsint64Number of times the post was crossposted.
subredditstringName of the subreddit.
subreddit_idstringUnique ID of the subreddit.
subreddit_subscribersint64Number of members in the subreddit.
domainstringThe domain of the link submitted (if any).
crosspost_parentstringID of the parent post if this is a crosspost.

💬 Comments

FieldTypeDescription
authorstringThe Reddit username of the commenter.
namestringUnique base36 ID of the comment (e.g., t1_xxx).
bodystringThe text content of the comment.
created_utcstringDate of creation (YYYY-MM-DD).
scoreint64Net upvotes minus downvotes.
controversialityint64Reddit's controversiality metric (0 or 1).
parent_idstringThe ID of the comment/post this is replying to.
link_idstringThe ID of the parent submission.
subredditstringName of the subreddit.
subreddit_idstringUnique ID of the subreddit.

💻 How to Use

You can easily load this dataset using the Hugging Face datasets library. Since it's stored in Parquet, it will stream efficiently.

python
from datasets import load_dataset

# Load the dataset (Streaming mode is recommended due to massive size)
dataset = load_dataset("anhchanghoangsg/reddit_pushshift_dataset_cleaned", streaming=True)

# Print the first row of the train split
print(next(iter(dataset['train'])))
anhchanghoangsg/reddit_pushshift_dataset_cleaned · CoolFace