open-index/arctic

Arctic Shift Reddit Archive

Every Reddit comment and submission since 2005, organized as monthly Parquet shards

Table of Contents

• —What is it?
• —What is being released?
• —Breakdown by type and year
• —How to download and use this dataset
• —Dataset statistics
• —Monthly breakdown
• —Dataset card
• —Dataset summary
• —Dataset structure
• —Dataset creation
• —Considerations for using the data
• —Additional information

What is it?

The full Reddit archive from Arctic Shift, converted to Parquet and hosted here for easy access. Covers every public subreddit from 2005-12 through 2026-02.

Right now the archive has 15.7B items (12.9B comments, 2.8B submissions) in 1.3 TB of compressed Parquet. Comments and submissions are stored as separate datasets, split into monthly shards you can load individually or stream together.

Reddit has been around since 2005. Millions of people use it to talk about everything - programming, sports, cooking, politics, niche hobbies. That makes it one of the best sources of natural conversation data for language model training, sentiment analysis, community research, and information retrieval. Most Reddit datasets only cover specific subreddits or time windows. This one covers all of it.

What is being released?

Monthly Parquet files, split by type (comments vs submissions). Small months fit in one shard. Large months (post-2015 or so) get split into multiple ~200 MB shards.

data/
  comments/
    2005/12/000.parquet       earliest month with data
    2006/01/000.parquet
    ...
    2023/06/000.parquet
              001.parquet     large months get multiple shards
              002.parquet
  submissions/
    2005/12/000.parquet
    2006/01/000.parquet
    ...
stats.csv                     one row per committed (month, type) pair
zst_sizes.json                .zst file sizes for all months (from torrent metadata)
states.json                   live pipeline state (updated every ~5 min)

stats.csv tracks every committed (month, type) pair with row count, shard count, Parquet size, original .zst size, processing time, and commit timestamp. zst_sizes.json maps every (type, month) pair to its .zst archive size — useful for estimating remaining work.

Breakdown by type and year

Comments

  2005  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  1.1K
  2006  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  417.2K
  2007  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  2.5M
  2008  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  7.2M
  2009  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  18.9M
  2010  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  48.5M
  2011  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  123.3M
  2012  ███░░░░░░░░░░░░░░░░░░░░░░░░░░░  260.3M
  2013  ████░░░░░░░░░░░░░░░░░░░░░░░░░░  402.2M
  2014  ██████░░░░░░░░░░░░░░░░░░░░░░░░  531.8M
  2015  ███████░░░░░░░░░░░░░░░░░░░░░░░  667.8M
  2016  █████████░░░░░░░░░░░░░░░░░░░░░  799.9M
  2017  ███████████░░░░░░░░░░░░░░░░░░░  972.9M
  2018  ██████████████░░░░░░░░░░░░░░░░  1.2B
  2019  ███████████████████░░░░░░░░░░░  1.7B
  2020  █████████████████████████░░░░░  2.2B
  2021  ██████████████████████████████  2.6B
  2022  ████████████████░░░░░░░░░░░░░░  1.4B

Submissions

  2005  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  5.4K
  2006  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  304.7K
  2007  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  908.2K
  2008  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  2.5M
  2009  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  4.8M
  2010  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  6.9M
  2011  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  15.0M
  2012  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  29.3M
  2013  ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░  39.7M
  2014  ███░░░░░░░░░░░░░░░░░░░░░░░░░░░  52.9M
  2015  ████░░░░░░░░░░░░░░░░░░░░░░░░░░  70.5M
  2016  █████░░░░░░░░░░░░░░░░░░░░░░░░░  90.6M
  2017  ███████░░░░░░░░░░░░░░░░░░░░░░░  117.2M
  2018  █████████░░░░░░░░░░░░░░░░░░░░░  152.4M
  2019  ██████████████░░░░░░░░░░░░░░░░  235.7M
  2020  ███████████████████░░░░░░░░░░░  326.7M
  2021  ██████████████████████░░░░░░░░  371.0M
  2022  ██████████████░░░░░░░░░░░░░░░░  234.1M
  2023  ██░░░░░░░░░░░░░░░░░░░░░░░░░░░░  39.2M
  2024  ████████████████████████████░░  477.4M
  2025  ██████████████████████████████  494.3M
  2026  █████░░░░░░░░░░░░░░░░░░░░░░░░░  87.5M

How to download and use this dataset

Load comments or submissions separately, filter by year or month, or stream the whole thing. Standard Hugging Face Parquet layout, works with DuckDB, datasets, pandas, and huggingface_hub out of the box.

Using DuckDB

DuckDB reads Parquet directly from Hugging Face - no download step needed.

-- Top 20 subreddits by comment volume (all time)
SELECT subreddit, count(*) AS comments
FROM read_parquet('hf://datasets/open-index/arctic/data/comments/**/*.parquet')
GROUP BY subreddit
ORDER BY comments DESC
LIMIT 20;

-- Monthly submission volume for 2023
SELECT
    strftime(created_at, '%Y-%m') AS month,
    count(*) AS submissions,
    sum(num_comments) AS total_comments
FROM read_parquet('hf://datasets/open-index/arctic/data/submissions/2023/**/*.parquet')
GROUP BY month
ORDER BY month;

-- Most active authors across all comments
SELECT author, count(*) AS comments, avg(score) AS avg_score
FROM read_parquet('hf://datasets/open-index/arctic/data/comments/**/*.parquet')
WHERE author != '[deleted]'
GROUP BY author
ORDER BY comments DESC
LIMIT 20;

-- Average comment length by year
SELECT
    extract(year FROM created_at) AS year,
    avg(body_length) AS avg_length,
    count(*) AS comments
FROM read_parquet('hf://datasets/open-index/arctic/data/comments/**/*.parquet')
GROUP BY year
ORDER BY year;

-- Top linked domains in submissions
SELECT
    regexp_extract(url, 'https?://([^/]+)', 1) AS domain,
    count(*) AS posts
FROM read_parquet('hf://datasets/open-index/arctic/data/submissions/**/*.parquet')
WHERE url IS NOT NULL AND url != ''
GROUP BY domain
ORDER BY posts DESC
LIMIT 20;

Using datasets

from datasets import load_dataset

# Stream all comments without downloading everything
comments = load_dataset("open-index/arctic", "comments", split="train", streaming=True)
for item in comments:
    print(item["author"], item["subreddit"], item["body"][:80])

# Load submissions for a specific year
subs = load_dataset(
    "open-index/arctic", "submissions",
    data_files="data/submissions/2023/**/*.parquet",
    split="train",
)
print(f"{len(subs):,} submissions in 2023")

Using huggingface_hub

from huggingface_hub import snapshot_download

# Download only 2023 comments
snapshot_download(
    "open-index/arctic",
    repo_type="dataset",
    local_dir="./arctic/",
    allow_patterns="data/comments/2023/**/*",
)

For faster downloads, install pip install huggingface_hub[hf_transfer] and set HF_HUB_ENABLE_HF_TRANSFER=1.

Using the CLI

# Download a single month of submissions
huggingface-cli download open-index/arctic \
    --include "data/submissions/2024/01/*" \
    --repo-type dataset --local-dir ./arctic/

Dataset statistics

Monthly breakdown

<details> <summary>Click to expand full monthly table (199 comment months + 227 submission months)</summary>

</details>

Query per-month stats directly:

SELECT year, month, type, shards, count, size_bytes
FROM read_csv_auto('hf://datasets/open-index/arctic/stats.csv')
ORDER BY year, month, type;

stats.csv columns:

Dataset card for Arctic Shift Reddit Archive

Dataset summary

A repackaging of the Arctic Shift monthly Reddit dumps into Parquet. Arctic Shift re-processes the PushShift Reddit archive, which captured most public Reddit content from the early days through the 2023 API changes.

Covers every public subreddit, every month, both comments and submissions. Built for research, analysis, and training. People use it for:

• —Language model pretraining and fine-tuning - one of the largest sources of natural conversation on the internet
• —Sentiment and trend analysis - two decades of public opinion on just about everything
• —Community research - thousands of subreddits, each with its own culture and moderation norms
• —Information retrieval - real questions and answers from r/AskReddit, r/explainlikeimfive, and others
• —Content moderation research - moderation signals are preserved in the data

Dataset structure

Data instances

Example comment:

{
  "id": "c0001",
  "author": "spez",
  "subreddit": "reddit.com",
  "body": "Welcome to Reddit!",
  "score": 42,
  "created_utc": 1134028003,
  "created_at": "2005-12-08T10:06:43",
  "body_length": 19,
  "link_id": "t3_17",
  "parent_id": "t3_17",
  "distinguished": null,
  "author_flair_text": null
}

Example submission:

{
  "id": "abc123",
  "author": "kn0thing",
  "subreddit": "reddit.com",
  "title": "The Downing Street Memo",
  "selftext": "",
  "score": 15,
  "created_utc": 1118895720,
  "created_at": "2005-06-16T01:02:00",
  "title_length": 23,
  "num_comments": 3,
  "url": "http://www.timesonline.co.uk/...",
  "over_18": false,
  "link_flair_text": null,
  "author_flair_text": null
}

Data fields

Comments (data/comments/YYYY/MM/NNN.parquet)

Submissions (data/submissions/YYYY/MM/NNN.parquet)

Data splits

Two named configs: comments and submissions. Each loads all monthly shards as a single train split.

You can also load specific years or months with data_files:

# Load just January 2020 comments
ds = load_dataset("open-index/arctic", data_files="data/comments/2020/01/*.parquet", split="train")

# Load all 2023 submissions
ds = load_dataset("open-index/arctic", data_files="data/submissions/2023/**/*.parquet", split="train")

Dataset creation

Why we built this

Reddit is one of the best sources of real human conversation on the internet, but getting at the full archive got a lot harder after Reddit locked down API access in 2023. The Arctic Shift project preserves the data as monthly .zst JSONL dumps. We convert those dumps to Parquet on Hugging Face so you can query with DuckDB, stream with datasets, or bulk download without any special tooling.

Source data

Everything comes from Arctic Shift torrent archives, which re-process the PushShift Reddit dumps. Source format is .zst-compressed JSONL, one JSON object per line.

• —2005-12 through 2023-12: From the Arctic Shift bundle torrent
• —2024-01 onward: Individual monthly torrents from Arctic Shift

Processing steps

The pipeline is written in Go and uses DuckDB for the Parquet conversion. For each (month, type) pair:

1. 1.Download the .zst via BitTorrent with selective file priority (only the needed file from the bundle, not the whole archive)
2. 2.Stream through a klauspost/compress zstd decoder with a 2 GB window
3. 3.Chunk the JSONL into ~2 million line batches, writing each to a temp file
4. 4.Convert each chunk to Parquet with DuckDB read_json_auto, explicit column selection, TRY_CAST, Zstandard compression, 131K-row row groups
5. 5.Delete each temp chunk right after the shard is written (disk is tight)
6. 6.Commit all shards plus updated stats.csv and README.md to Hugging Face
7. 7.Clean up local shards after the commit goes through

The pipeline picks up where it left off - stats.csv tracks what has been committed, and those pairs get skipped on restart. Disk usage stays minimal: at most one .zst, one JSONL chunk, and the current month's shards on disk at a time.

No filtering, deduplication, or content changes. The data matches the Arctic Shift dumps exactly. All Parquet files use Zstandard compression.

Personal and sensitive information

Usernames and user-generated text are included as they appeared publicly on Reddit. Deleted accounts show as [deleted], deleted content as [removed].

No PII scrubbing has been done. At this scale, the dataset almost certainly contains personal information that people posted publicly. If you find something that should be removed, open a discussion on the Community tab.

Considerations for using the data

Social impact

Making the full Reddit archive accessible in a standard format should help researchers study how online communities work, how language changes over time, and how one of the internet's biggest platforms has shaped public discourse.

Biases

Reddit skews young, male, English-speaking, and North American/European. Subreddits vary wildly in culture, moderation, and toxicity. The voting system amplifies what each community already agrees with.

We did not filter, score, or assess the data in any way. Controversial, toxic, and NSFW content is all in there. Apply your own filtering for your use case.

Known limitations

• —Completeness depends on PushShift. PushShift missed some content, especially in the earliest months and during ingestion outages.
• —Scores are snapshots. The score field is whatever PushShift captured at the time, not the final score.
• —Deleted content. Posts deleted before PushShift got to them are gone. Posts deleted after capture may still have the original text.
• —No user profiles. Just posts and comments. No karma, no account metadata.
• —Markdown and HTML. Comment bodies use Reddit's Markdown variant. Some old content has raw HTML.

Additional information

Licensing

Reddit content is subject to Reddit's Terms of Service. Arctic Shift distributes the archive under permissive research terms. This repackaging is provided as-is for research and education.

Not affiliated with or endorsed by Reddit, Inc. or Arctic Shift.

Thanks

All the data here comes from Arctic Shift, which preserves and distributes the PushShift Reddit archive through Academic Torrents. None of this would be practical without their work.

Contact

Questions, feedback, or issues - open a discussion on the Community tab.

Last updated: 2026-08-06 06:59 UTC