awesomedudeworld/helios

Helios — AI Solar Flare Predictor & an Honest-Evaluation Study

A live solar-flare forecasting system and the instrument for a research finding: standard benchmark scores overstate the real-time operational skill of ML flare forecasts — by up to 2×. Helios learns from SHARP magnetic-field data, classifies what the Sun is doing right now from live NOAA data, and forecasts flare probability at 12 / 24 / 48 h lead times — with an escalating warning for X-class events. Ships with a polished web forecast page.

The research result (PAPER.md · abstract · reproduce it): a model scoring TSS 0.77 on the SWAN-SF benchmark drops to TSS 0.35–0.64 on live out-of-sample JSOC data across three solar-cycle phases — above the live point estimate everywhere, above the cluster-bootstrap 95% CI in two of three periods. A controlled 2×2 shows retraining on live data does not close the gap (it is a property of the benchmark's evaluation set); the recoverable loss is the threshold's objective — re-tuning it for TSS instead of F1, using only benchmark validation data, recovers most of the skill (live recalibration adds nothing further). Every benchmark number below should be read with that caveat — that is the point of the paper.

Deployed models: live SHARP (JSOC-trained RandomForest, benchmark TSS 0.77 / honest live TSS 0.35–0.64 by period), the SWAN-SF benchmark model (ExtraTrees, benchmark TSS 0.87), and an OMNI-trained geomagnetic-storm model (TSS 0.47). Three selectable operating points trade recall for precision — see Operating points.

The robustness companion (dashboard scorecard)

Alongside the paper's research/ pipeline, python -m solarflare.reproduce regenerates a second, complementary evidence set — same research question, independent protocol — and writes RESULTS.md:

• —the 2×2 across four operational years (2013/15/16/17) with cluster-bootstrap 95% CIs and frozen-threshold deployment scores;
• —a storm-model generalisation check (the same benchmark optimism appears in a different feature space — it is not a flare quirk);
• —a mechanism diagnosis (label audit via the HARP→NOAA mapping, distribution-shift KS tests, permutation-importance divergence);
• —a comparison against NOAA's own archived official forecasts (RSGA warehouse) on identical days;
• —a growing prospective forecast record (forecasts verified after the fact);
• —a dose–response curve (does the gap shrink as live training data grows? python -m solarflare.experiments.dose_response);
• —self-correcting deployment — python -m solarflare.recalibrate saves an "operational" operating point (threshold recalibrated on one year of live data) into a model artifact; opt in live via sharp_live.operating_point: operational. The scorecard shows the corrected-deployment rows.

Why two pipelines, and why their numbers differ. These are deliberately independent replications of the same finding, not the same computation twice — a judge who compares them should know the mapping:

Both apply the same fail-closed label-quality gate (2023 excluded in both), so their year sets differ only by which years each protocol scores — the headline conclusion (benchmark overstates; live training doesn't close it; the fix is the TSS threshold) is identical across both. Numbers differ because the thresholds and year sets differ, not because the pipelines disagree.

The label-attribution decay behind the 2023 exclusion is measured across years in config.yaml (scorecard.label_attribution_by_year): healthy 0.94–0.98 through 2017, then 0.55 (2021), 0.11 (2022), 0.15 (2023) — a curve, not a single bad year.

See MODEL_CARD.md for honest per-model documentation, and PREREGISTRATION.md for the pre-committed 2024 cross-cycle test. The dashboard's "Model skill scorecard" panel shows all of it live, with CIs, per-year chips, a reliability diagram (figures/fig8_calibration.png is its board-quality twin), and the NOAA comparison.

The one thing you must understand first

The training data and the live data are different physics, and the system is built to respect that instead of faking a connection:

A model trained on magnetic SHARP parameters cannot literally consume X-ray flux — they are different feature spaces. Pretending it could would be exactly the "cheating" the project brief warns against. So the predictor runs three independent tracks and ensembles whatever is available, always returning a clean forecast even if a source is down.

                       ┌─────────────────────────────┐
  NOAA GOES X-ray ───► │ NOWCAST  (exact, no ML)      │ ─► current class + X-WARNING
   (always live)       └─────────────────────────────┘
                       ┌─────────────────────────────┐
                  ───► │ FLUX forecast (persistence) │ ─┐
                       └─────────────────────────────┘  │
  NOAA regions    ───► │ NOAA official region forecast│ ─┼─► ENSEMBLE ─► 12/24/48h
                       ┌─────────────────────────────┐  │
  SWAN-SF model   ───► │ SHARP ML  (when fed magnetic)│ ─┘
                       └─────────────────────────────┘

Because a flare's class is defined by its GOES 1–8 Å peak flux, identifying a flare happening now needs no ML at all — it's a measurement. The machine learning is reserved for forecasting the future, which is the honest place for it.

Quickstart

# 1. install  (requirements.txt = loose ranges; requirements.lock = exact
#    pinned versions this project's results were produced with — use the lock
#    to reproduce the paper's numbers bit-for-bit)
pip install -r requirements.txt        # or: pip install -r requirements.lock

# 2. prove the whole pipeline works (NO download, NO GPU needed)
python scripts/smoke_test.py
#    -> trains on a synthetic fixture in a TEMP dir (your real model in
#       models/ is untouched), evaluates skill scores, then hits live NOAA.

# 3. run the API + open the forecast page
python -m uvicorn api.server:app --port 8000
#    -> open http://127.0.0.1:8000/

Train on the real SWAN-SF data

python -m solarflare.download      # pulls the .pkl partitions (Google Drive)
python -m solarflare.train         # trains on partitions 1-3, tests on 5

Training writes three artifacts to models/:

Rebuilding the research datasets (fresh clone — read before reproduce)

Honest cost note: python -m solarflare.reproduce regenerates every research artifact from datasets that are gitignored. A fresh clone must rebuild them first — external downloads plus several hours of JSOC queries. It is "one command" only once the datasets below exist in data/sharp_live/.

# 1. SWAN-SF benchmark partition (manual): download partition1_instances.tar.gz
#    from Harvard Dataverse doi:10.7910/DVN/EBCFKM (~1 GB) into data/swansf_full/,
#    then convert (~minutes):
python -m solarflare.swansf_data --tars data/swansf_full/partition1_instances.tar.gz \
    --out data/sharp_live/dataset_swansf_p1.npz

# 2. JSOC operational years (network-heavy: ~15-40 min per year):
python -m solarflare.sharpdata --start 2013-01-01 --end 2014-01-01 --out data/sharp_live/dataset_2013.npz
#    ... repeat for 2011, 2012, 2014, 2015, 2016, 2017 (2023 exists but is
#    excluded from label-dependent scoring — see scorecard.label_attribution_by_year
#    in config.yaml; scripts/label_attribution.py measures the rates).

# 3. Cleaned SWAN-SF .pkl partitions for the benchmark model (Google Drive, ~1.2 GB):
python -m solarflare.download

Expected shapes (sanity check — window arrays are (n, 60, 17); counts within a few % of these mean your build matches ours):

How it avoids "cheating"

• —Leakage-free split — trains on the augmented partitions, tunes the decision threshold on a separate validation partition, and reports final numbers on a clean test partition the model never saw. This is the standard SWAN-SF protocol.
• —Honest metrics — accuracy is useless when 95% of samples are "no-flare", so the headline is TSS (True Skill Statistic = recall − false-alarm rate; 0 = no skill) and HSS (Heidke Skill Score). See solarflare/evaluate.py.
• —Class imbalance handled with balanced sample weights, not by resampling the test set.
• —Calibrated probabilities — isotonic calibration so a "21%" forecast means roughly 21%.

Operating points (recall vs. precision)

Flares are ~1% of samples, so you can't have high recall and high precision — you pick a trade-off. Training fits all three on validation and reports each on the clean test partition (ExtraTrees winner):

What's a "good" precision here? Above the 1.3% base rate = skill. Research optimises TSS (precision 5–15%); operational alerting wants ~30–50%. On this task the realistic ceiling while staying useful is ~34% (high_precision), still catching ~70% of flares. Select the point in config.yaml (live.operating_point). The UI also shows a probability band (Low/Moderate/High/Severe) to sidestep the hard threshold entirely.

Flare taxonomy (matches the brief)

Driven entirely by config.yaml:

The default trainable target is "any M-or-greater flare in the next 24 h" (the Cleaned-SWANSF labels are binary). To get a 4-way no-flare/C/M/X model, supply the original SWAN-SF multi-class labels and set training.task: multiclass in config.yaml — the same pipeline handles it.

API

The space-weather platform (Parts 1 & 2)

Beyond the flare forecast, Helios layers on a geographic hazard map, an orbital view, a real geomagnetic-storm model, and a reproducible skill experiment — each in its own feature space (the cardinal rule: never mix the physics).

Why separate hazard layers? Solar X-rays drive flares and the dayside HF blackout; the Sun's magnetic field (SHARP) drives flare prediction; Earth's field + the L1 solar wind drive geomagnetic storms at high latitudes. Different feature spaces — never mixed. The map shows each as its own toggleable layer.

Geomagnetic-storm model (the scientific core)

solarflare/storm.py (+ stormdata.py) trains on NASA OMNI hourly data (CDAWeb HAPI): a 24 h window of L1 solar-wind drivers (Bz, speed, density, dynamic pressure, IMF clock angle, Newell coupling) → P(a G1+ storm, Kp ≥ 5, in the next 24 h). Same honest protocol as the flare model — a leakage-free chronological split with a multi-day gap (never shuffled), isotonic-calibrated probabilities, three operating points, TSS/HSS via evaluate.py. Live inference rebuilds the features from the real-time L1 feeds, with NOAA's Kp forecast as corroboration and a climatology fallback.

python -m solarflare.storm               # train on real OMNI (HAPI)
python -m solarflare.storm --synthetic   # offline fixture (no download)

Lead-time vs skill experiment

solarflare/experiments/leadtime_skill.py forecasts the same storm target from several vantage points and plots skill (TSS) vs forecast lead time: climatology (floor), persistence, an L1 logistic, 27-day recurrence, and a CME track (NASA DONKI). Leakage-free; deterministic with a fixed seed:

python -m solarflare.experiments.leadtime_skill

Outputs land in solarflare/experiments/results/ (CSV, JSON, a publication PNG, and RESULTS.md) and the figure shows in the dashboard. Headline finding: no single vantage point forecasts storms well at every lead — L1 wins at short lead, CMEs extend to 1–3 days, and only a weak 27-day recurrence reaches further out.

Impacts, aurora & alerts

• —Impact statements (/api/impact) map NOAA's R/S/G scales to plain language.
• —Aurora visibility ("how far south") is derived from Kp on /api/geomag.
• —Threshold alerts (/api/alerts, solarflare/alerts.py) always log, POST to a webhook if alerts.webhook_url or $ALERT_WEBHOOK_URL is set, and keep email off by default. No secrets are committed.

Resilience / failsafes

Every live fetch (solarflare/sources.py):

1. 1.tries multiple URLs in order (primary → secondary → shorter feed),
2. 2.has a hard timeout,
3. 3.caches each success to .cache/ (the failsafe backup),
4. 4.falls back to the most recent cache if all sources fail, clearly flagged "cached / N min old",
5. 5.if everything is down, the predictor returns a climatological fallback forecast plus a notice — it never errors out.

Live sources used: NOAA SWPC GOES X-ray (primary + secondary + 6-hour), NOAA Solar Region Summary, NOAA GOES flare events, and NASA DONKI (optional key). Add your free key at <https://api.nasa.gov> in config.yaml.

Make it better (it's a first model on purpose)

• —Train on the real SWAN-SF partitions (download → train).
• —Add multi-class labels for true upper-C / M / X separation.
• —Wire live SHARP from JSOC/SDO so the ML track runs in production (predictor.sharp_forecast already accepts a feature vector).
• —Swap HistGradientBoosting for LightGBM/XGBoost or an LSTM in solarflare/train.py — the data + eval harness stay the same.
• —Tune the flux-persistence priors in solarflare/fluxmodel.py against the GOES flare catalogue.

Honest limitations

• —Solar-flare forecasting has a hard skill ceiling; no model gets near "certainty". Treat outputs as probabilistic guidance alongside official NOAA SWPC products.
• —The always-on web forecast runs the flux + NOAA tracks; the SHARP ML track activates only when magnetic features are supplied (the synthetic smoke model proves the plumbing).
• —DEMO_KEY for NASA DONKI is heavily rate-limited — add your own key.

File map

config.yaml                 all thresholds, paths, sources, training, experiment & alert options
solarflare/
  config.py    labels.py    config loader · flare taxonomy + X-warnings
  data.py      train.py     SWAN-SF loader + features · flare trainer (.pkl/.joblib)
  evaluate.py  download.py  TSS/HSS skill scores · dataset downloader
  sources.py   nowcast.py   resilient live fetchers · current-state from flux
  fluxmodel.py predictor.py persistence forecaster · orchestrator + fallbacks
  hazard.py    geomag.py    dayside HF blackout (D-RAP) · auroral oval + Kp (OVATION)
  satellites.py             satellites-at-risk by altitude band (CelesTrak TLE)
  storm.py     stormdata.py geomagnetic-storm forecaster · OMNI loader + features
  impact.py    alerts.py    R/S/G plain-language impacts · threshold alerts (+ demo)
  notify.py                 daily/alert prediction logger + verifier · prediction_history.csv exporter
  sharpdata.py sharptrain.py JSOC+HEK dataset builder · live-model trainer
  swansf_data.py sharp_live.py SWAN-SF tar parser · live JSOC inference
  scorecard.py storm_scorecard.py  the 2x2 research experiment (+CIs) · storm check
  reproduce.py             one-command reproduction of every research artifact
  recalibrate.py           self-correcting deployment (operational op point)
  train_variant.py         train a deployable model variant (own artifact)
  experiments/dose_response.py     gap vs. amount of live training data
  experiments/leadtime_skill.py    lead-time vs skill experiment (CSV/PNG/RESULTS.md)
  experiments/gap_diagnosis.py     why the gap exists (labels/shift/importance)
  experiments/noaa_baseline.py     deployed model vs NOAA's archived official forecast
api/server.py                FastAPI backend + serves the frontend
frontend/index.html          the forecast page (dark ops dashboard + map/globe)
scripts/smoke_test.py        end-to-end proof with no download
scripts/regression_test.py   read-only guard run after every change