Roronotalt/bluesky-ten-million

Ten Million bluesky posts

<!-- Provide a quick summary of the dataset. -->

This dataset contains 5 million public posts collected from Bluesky Social's firehose API, intended for machine learning research and experimentation with social media data.

This dataset was inspired by the Alpindales original 2 million posts dataset, this dataset expands on that dataset with much more data.

Alpins dataset did not get author handles or image urls & metadata that was included in the posts. The images and their captions could potenically be invaluble for training so they have been collected.

This is the small version of the dataset to come for testing with formatting/smaller projects.

This dataset is my own and is unaffiliated with bluesky or any potential employer.

Dataset Structure

<!-- Provide a longer summary of what this dataset is. -->

• —Curated by: Roro
• —License: MIT

Uses

<!-- Address questions around how the dataset is intended to be used. -->

The dataset could be used for:

• —Study social media trends
• —Research on social media content moderation
• —Studying conversation structures and reply networks

I have not been able to figure out how to parse the atproto image ref bytes into a image or blob url. I would appreciate a PR for that.

The dataset is meant to be downloaded with the huggingface load_dataset() function. From there you can either run the dataset as a iterable stream so you do not have to worry about memory or you can convert to a pandas dataframe.

Note that you will need the to install the following libraries:

pip install pandas pyarrow datasets huggingface_hub

To download/load the huggingface dataset:

from datasets import load_dataset
dataset = load_dataset("Roronotalt/bluesky", split="train")

To pandas:

new_dataset = dataset.to_pandas()

You can then save the pandas dataframe as a csv.

Alternativley if you download the provided dataset parquet file in /data, you can convert the file to a csv using the following python code:

python -c "import pandas as pd;
df = http://pd.read_parquet('train-0000.parquet', engine='pyarrow');
http://df.to_csv('output_file.csv', index=False) 
"

Credit to @TyrantsMuse on twitter for the code snippet

Dataset Curation

The dataset not is filtered, sorting the dataset for quality or moderation may make it more valuable for your use cases. The dataset is as-is and no liablity is provided.

Deduping was done based on the post URIs. The dataset is sorted by the author column.