emperor-mew/voidly-bench-v1

Voidly Benchmark v1

A public benchmark for the internet-censorship-forecasting task. Labeled incidents, joined raw evidence, per-source reliability, and a reproducibility script that lands the same F1 / AUC numbers the Voidly Sentinel model publishes at /v1/sentinel/accuracy.

If your model claims to predict internet shutdowns, this is the dataset to prove it on.

What's in here

The task

For each (country, date) pair, predict whether at least one incident will start within the following 7 days. Features: lagged OONI-derived block rates (1/2/3/7/14d), rolling mean/std over 7/14/30d, GDELT protest article counts, IODA outage alerts, OONI anomaly counts, election/protest proximity, historical incident density, and country risk tier.

Primary metrics: ROC AUC and F1 at the F1-optimal threshold.

Targets on this released dataset (what reproduce.py should hit within ±2pp):

Prod Sentinel model, same algorithm but with the full GDELT + social-signals feature set (not in this snapshot):

The benchmark F1 is higher than prod because the auxiliary features GDELT + social-signals + probe metrics carry noise that the cleaner OONI-only benchmark avoids. Published papers should state which feature set they used. reproduce.py is deterministic; check seed + data SHA256 before claiming a delta.

How the labels were built

• —OONI: webconnectivity, signal, whatsapp, telegram, facebookmessenger, tor, httpinvalidrequestline, httpheaderfieldmanipulation — rows with anomaly = 1.
• —IODA: ASN-level outage events with score above the IODA alert threshold.
• —CensoredPlanet: Satellite (DNS) and Hyperquack (HTTP/HTTPS) blocking events with blocked = True.
• —Incident formation: clustering of evidence by (country, week, domain family) with a z-score spike detector over 30-day baselines. Status confirmed_censorship = confidence ≥ 0.7 AND ≥ 2 sources agreeing.

Full pipeline: <https://github.com/voidly-ai/voidly/tree/main/scripts> (scripts: ingest-ooni-*.py, ingest-ioda.py, ingest-censoredplanet-*.py, incident-labeler.py, dedup-incidents.py).

Per-source reliability

Reliability is measured as confirmed_linked / linked_incidents — the fraction of incidents that a given source's evidence rows contribute to that end up at confidence ≥ 0.7. Current values:

IODA's high precision is an artifact of its alert-threshold cutoff — IODA only emits when ASN-level reachability drops significantly, which is rare and usually real. OONI's low precision reflects its breadth: many measurements flag as anomalies for reasons unrelated to censorship (test-server issues, client misconfig, transient outages). Use these as instance weights in training if you want to down-weight noisier sources.

Known gaps (don't publish results that pretend these don't exist)

• —Positive base rate is ~7%; splits need stratification or results are noise.
• —Temporal leakage: splitting randomly inflates AUC. Use time-based splits (train on dates ≤ T, evaluate after T) for honest numbers.
• —Country effects dominate: leave-country-out AUC drops from 0.98 to ~0.55 with the reference features. The model is learning per-country base rates as much as temporal structure.
• —Ingestion gaps: Apr 2026 has an OONI gap Mar 31 - Apr 10 on this dataset snapshot. Features computed over that window will have missing-as-zero artifacts.
• —Probes not yet in high-risk countries: probe_metrics covers US/CA/DE/IN/FR/BR at this snapshot, not IR/CN/RU. Probe-derived features are zero in the published dataset.

Licensing

Data: CC BY 4.0 (attribution: "Voidly — https://voidly.ai"). Code in reproduce.py: Apache 2.0.

Derived sources retain their upstream licenses — OONI, IODA, CensoredPlanet each publish their raw measurements under open terms; this dataset aggregates and labels them. Upstream citations required when publishing derived work:

• —OONI: <https://ooni.org/> (MIT / CC BY-NC-SA for some aggregates)
• —IODA: <https://ioda.inetintel.cc.gatech.edu/> (research use)
• —CensoredPlanet: <https://censoredplanet.org/> (research use)

Reproducibility

pip install pandas pyarrow scikit-learn xgboost
python reproduce.py

# Expected output (within ±2pp):
#   rows_used ≈ 14300
#   AUC      ≈ 0.98
#   F1       ≈ 0.92
#   precision ≈ 0.94
#   recall   ≈ 0.89
#   threshold ≈ 0.23
#   within_tolerance: True

Run time: ~30 minutes on 6 CPU cores. Memory: <2 GB.

Artifact hashes and seeds:

• —incidents.parquet — sha256 will be stamped on HF upload
• —evidence.parquet — sha256 will be stamped on HF upload
• —np.random.seed(42) used throughout reproduce.py

Citation

@dataset{voidly_bench_v1_2026,
  author    = {Voidly},
  title     = {Voidly Benchmark v1: Labeled Internet Censorship Incidents
               with Joined Evidence},
  year      = {2026},
  publisher = {HuggingFace},
  license   = {CC BY 4.0},
  url       = {https://huggingface.co/datasets/emperor-mew/voidly-bench-v1}
}

Companion paper in preparation — this card will be updated with the NeurIPS Datasets & Benchmarks submission link.

Changelog

• —v1 (2026-04-17): initial release. 1,574 incidents, 49,596 evidence rows, 3 sources. Reference model: GradientBoosting, AUC 0.98 / F1 0.79.