Datasetpublic

sentence-transformers/reddit-title-body

Reddit (Title, Body)-Pairs This dataset contains jsonl-Files about (title, body) pairs from Reddit. Each line is a JSON object of the following format: {'title': 'The title of a thread', 'body': 'The longer body of the thread', 'subreddit': 'subreddit_name'} The 2021 file contains submissions up until including 2021-06. Entries in the respective files are shuffled on a monthly basis. The data has been filtered for: Remove threads with an upvote_ratio < 0.5 Only include threads… See the full description on the dataset page: https://huggingface.co/datasets/sentence-transformers/reddit-title-body.

sourceHugging Faceupdated 5y agoView on Hugging Face

28likes932downloads

Dataset Card

Reddit (Title, Body)-Pairs

This dataset contains jsonl-Files about (title, body) pairs from Reddit. Each line is a JSON object of the following format:

{'title': 'The title of a thread', 'body': 'The longer body of the thread', 'subreddit': 'subreddit_name'}

The 2021 file contains submissions up until including 2021-06. Entries in the respective files are shuffled on a monthly basis.

The data has been filtered for:

—Remove threads with an upvote_ratio < 0.5
—Only include threads with a title more than 25 characters and bodies with len(title)+25 < len(body) < 4096
—Only keep threads with at least 3 comments or at least 3 upvotes.

Overview

File	Lines
reddittitletext_2010.jsonl.gz	431,782
reddittitletext_2011.jsonl.gz	1,673,264
reddittitletext_2012.jsonl.gz	3,727,526
reddittitletext_2013.jsonl.gz	5,713,956
reddittitletext_2014.jsonl.gz	8,538,976
reddittitletext_2015.jsonl.gz	11,064,453
reddittitletext_2016.jsonl.gz	12,224,789
reddittitletext_2017.jsonl.gz	13,558,139
reddittitletext_2018.jsonl.gz	15,552,110
reddittitletext_2019.jsonl.gz	19,224,970
reddittitletext_2020.jsonl.gz	23,030,988
reddittitletext_2021.jsonl.gz	12,704,958

Note: The data comes from Pushshift. Please have a look at the respective license of Reddit and Pushshift before using the data.

Be aware that this dataset is not filtered for biases, hate-speech, spam, racial slurm etc. It depicts the content as it is posted on Reddit.