CoolFace
Datasetpublic

WitchesSocialStream/AozoraDivr

Daily bleeding-edge snapshots of everything that came down the public Bluesky firehose, minimally groomed but absolutely NOT privacy-scrubbed. Treat it like you just tapped the fiber yourself. TL;DR What you get: Real-time posts, replies, likes, follows, blocks, new accounts, profile blobs—pretty much verbatim. What we stripped: Only literal garbage (malformed records, Mastodon mirror spam, bridge-bot junk). What we kept: Usernames, DIDs, PII, nudes, slurs, doxxes—all still… See the full description on the dataset page: https://huggingface.co/datasets/WitchesSocialStream/AozoraDivr.

sourceHugging Facecc-by-4.0updated 11h agoView on Hugging Face
5likes6.4kdownloads
Dataset Card

image/png

Daily bleeding-edge snapshots of everything that came down the public Bluesky firehose, minimally groomed but absolutely NOT privacy-scrubbed. Treat it like you just tapped the fiber yourself.

TL;DR

  • —What you get: Real-time posts, replies, likes, follows, blocks, new accounts, profile blobs—pretty much verbatim.
  • —What we stripped: Only literal garbage (malformed records, Mastodon mirror spam, bridge-bot junk).
  • —What we kept: Usernames, DIDs, PII, nudes, slurs, doxxes—all still inside.

Direct Use (good luck)

  • —Language-model pre-training at internet-scale.
  • —Memetic-propagation experiments.
  • —Adversarial red-team corpus.
  • —Anything that requires authentic, messy micro-blog data.

Out-of-Scope / Mis-use

  • —Shipping this as “safe-for-work”.
  • —Claiming “anonymized dataset” in IRB docs.
  • —Re-publishing unfiltered subsets.

Content Warnings (the NSFL edition)

What’s in here?FrequencyExample mitigation *you* need to write
Plain-text passwords/keys~0.001 %Grep-delete any `/^passwordpkcs8/i`
Self-harm imagery descriptions2–3 %Hard filter labels & keywords
CSAM pointers<0.0001 % but non-zeroHash match, auto-report
Extremist slogansSame as X/TwitterHave moderation queues ready
Full-res avatar/banner nudesLoadsBlur, crop, or drop blobs yourself

Dataset Layout

One line = one firehose repo#op. No joins done for you.

general.jsonl          → posts, replies (text, facets, labels, media CIDs)
interactions.jsonl     → likes | reposts | follows | blocks (+ `chg: create|delete`)
accounts.jsonl         → new accounts or bio/avi/banner updates
miracles.jsonl         → Everything listed as `Miracleroad!` (i.e. not a standard lexicon from bsky.)

Blob CIDs intact → fetch via https://bsky.social/xrpc/com.atproto.sync.getBlob?did=...&cid=...

Build Process

  1. 1.Connect firehose.
  2. 2.Drop malformed CBOR, external-federation junk, & Bridgy bot (HF forums leaks).
  3. 3.Push gzipped JSONL to GCS nightly.
  4. 4.No further filtering.

License & Citation

CC-BY-4.0. Fork, scrape, redistribute—just cite:

bibtex
@misc{bskyaozora-raw,
  title        = {AozoraDiving – Raw Bluesky Firehose},
  author       = {karakarawitch},
  howpublished = {\url{https://huggingface.co/datasets/WitchesSocialStream/bluesky-Aozora-Diving}},
  year         = 2024
}
WitchesSocialStream/AozoraDivr · CoolFace