wildwood77/bluesky-embeddings-daily

🛰️ Bluesky AI Analysis: Public Post Embeddings

This dataset contains vector embeddings of public posts from the Bluesky Social network, generated for the purpose of semantic search, discovery, and language model experimentation.

📦 Contents

Each row in the dataset includes:

• —uri: The AT URI of the post.
• —created_at: The full timestamp when the post was created.
• —created_date: The UTC calendar date (YYYY-MM-DD).
• —created_hour: The UTC hour of day (0–23).
• —text: The post's text content.
• —embedding: A 384-dimensional float vector representing the post's semantic content.
• —post_url: A link to the post on bsky.app.

The data is stored in Apache Parquet format for efficient querying and vector-based access.

📈 Embeddings

Embeddings are generated using `sentence-transformers/all-MiniLM-L6-v2`, a compact transformer model optimized for semantic similarity tasks. Each post is truncated to 300 characters prior to embedding.

🔁 Update Schedule

This dataset is updated twice daily, covering posts from the previous 3 days. New files may appear before old ones are consolidated.

📊 Use Cases

This dataset is suitable for:

• —Semantic search and nearest-neighbor lookup
• —Topic clustering
• —Prompt tuning and LLM finetuning (with care)
• —Language model evaluation on real-world social content

⚠️ Caveats

• —Public posts only: Only publicly visible Bluesky posts are included.
• —No personal data: The dataset does not include private user information or DMs.
• —Post volume: High-volume time slices may result in multiple files per day, eventually consolidated into daily files.
• —Embeddings only: This is not a full social graph or post archive — it's for semantic work.

🧪 Example Usage (Python)

import duckdb

con = duckdb.connect()
df = con.execute("SELECT * FROM 'posts-2025-07-14.parquet' LIMIT 5").fetchdf()
print(df[['text', 'embedding']])

🛠️ Source & Pipeline

This dataset is generated by the open-source project: 🔗 github.com/wildwood/bluesky-ai-analysis

📜 License

This dataset is made available under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. Use it freely, but please provide attribution.

For questions or suggestions, feel free to open an issue on the GitHub repo or contact the maintainer.