Arimancy/ercot-hourly-demand-weather

<!-- The double-brace placeholder facts below are rendered from buildmanifest.json by scripts/publishkaggle.py at stage time. Do not hand edit the numbers. -->

ERCOT Hourly Electricity Demand + Population-Weighted Weather (2015 to 2026)

A model-ready hourly time series for Texas grid load forecasting: ERCOT balancing-authority demand (MWh) pre-joined to population-weighted metro temperature, with calendar, holiday, and degree-hour features: 97,849 hours from 2015-07-01 to 2026-08-29, no missing hours, every imputed or preliminary value explicitly flagged.

[image]

The demand-temperature "U" above is the dataset's signature: mean demand bottoms out near 64 °F and climbs steeply with cooling load. The Jul 2026 record peak (91,075 MWh, about 99 °F weighted) anchors the right arm, Winter Storm Uri (Feb 2021) the left.

Why this dataset

The hard part of load forecasting isn't finding demand data — it's the join. This dataset does the error-prone work correctly and documents it:

• —UTC-canonical join. Demand and weather are joined strictly on UTC, which has no DST pathologies. Local time (America/Chicago) ships as a derived column; spring-forward days have 23 local hours and fall-back days 25, flagged via is_dst_transition_day rather than "fixed."
• —Population-weighted weather. Four metro ERA5 series (DFW, Houston, San Antonio, Austin) weighted by Census Vintage-2025 CBSA populations (0.3886 / 0.3623 / 0.1289 / 0.1201) — the same population-weighting idea EIA uses for degree days. El Paso is excluded because it is in WECC, not ERCOT.
• —Provenance you can filter on. The long backbone is Catalyst Cooperative's PUDL-cleaned EIA-930 (anomalies imputed, with reason codes); the freshest about 4 weeks comes straight from the EIA API and is flagged demand_source="eia_api" (preliminary, unimputed). Weather gap handling only ever interpolates interior gaps ≤ 3 hours, flagged. A shipped provenance.md, generated from the build manifest, records the exact source ranges, fetch timestamps, and file checksums.
• —Measured, not asserted. The bundled notebook ablates the weather join honestly: with strong demand lags already in the model, adding actual-weather features moves test MAPE by a small, quarter-dependent amount (pooled rolling-origin about 0.46 percentage points, helping in 5 of 7 test quarters and hurting in 2). The value here is the clean, reproducible join, the fresh hourly demand, and the ready-made degree features, not a headline forecast lift.

Sources

Methodology summary

Hourly ERCO demand (hour-ending UTC) from PUDL's cleaned parquet, topped up past PUDL's quarterly lag with the EIA API, deduplicated preferring PUDL. ERA5 2 m temperature pulled at each metro's airport grid point in GMT, combined as T(t) = Σᵢ wᵢ·Tᵢ(t) with fixed Vintage-2025 population weights (weighted value is NaN unless all four metros report). Left join on the UTC hour, reindexed to the complete hourly range, gaps flagged, short interior gaps (≤ 3 h) linearly interpolated and flagged. Calendar features computed from local wall-clock time. Holidays ship in two US federal semantics: statutory dates only (is_holiday_statutory; is_holiday is its exact alias, kept for backward compatibility) and observed (is_holiday_observed, weekend holidays also flagged on the adjacent observed weekday), plus federal and Texas combined (is_holiday_tx). Degree features are hourly degree-hours and daily degree-days, base 65 °F, from the weighted temperature. Full column-by-column reference: see the data dictionary.

Files

• —ercot_demand_weather_hourly_v1.parquet — recommended: typed, compressed, timezones survive round-trip.
• —ercot_demand_weather_hourly_v1.csv — universal access; timestamps are ISO 8601 with explicit offset.

Known limitations

• —The most recent about 4 weeks of demand is preliminary EIA data (demand_source="eia_api", unimputed).
• —The weather tail is incomplete and revisable: ERA5 publishes about 5 to 8 days behind real time (about 5 days in this build), so the newest rows have demand but null weather, flagged by temp_is_missing and never interpolated. Rows in roughly the last 2 to 3 months are preliminary near-real-time ERA5 (ERA5T) and may shift slightly when ECMWF finalizes a later release. On the local day that straddles the weather frontier, the daily degree-day columns are computed from a partial-day mean and broadcast to that day's hours, including the flagged null-weather ones.
• —ERA5 is a reanalysis (model) estimate, not station observations.
• —Four metros ≈ two-thirds of Texas population; rural load is implicitly under-weighted. ERCOT itself covers ~90% of Texas load.
• —Population weights are static (Vintage 2025).
• —A backtest on this data uses actual weather; do not compare its error head-to-head with ERCOT's live day-ahead forecast MAPE (~2%), which uses live weather forecasts and analyst correction.

Version history

• —v1.1 (2026-07): weather pinned to ERA5 across the whole history (v1 unintentionally served the API's Best Match blend); data refreshed through the build date; two explicit holiday semantics added (is_holiday_statutory, is_holiday_observed; is_holiday stays the statutory alias), schema now 29 columns; release hardening (a reason-predicated PUDL to EIA seam gate, decoupled demand and weather freshness gates, a shipped build provenance note with checksums, and release validation before any upload); the baseline notebook now measures the weather join's marginal value instead of attributing the whole model gain to it.
• —v1 (2026-07-16): initial public launch.

License and attribution

This dataset is published by Arimancy LLC under CC BY 4.0. The full license text and the attribution target also ship in the package as LICENSE.txt.

• —EIA-930 demand data: U.S. Energy Information Administration, Form EIA-930 / Hourly Electric Grid Monitor (US public domain). Source: U.S. Energy Information Administration (Aug 2026).
• —PUDL: cleaned/imputed EIA-930 demand via Catalyst Cooperative's Public Utility Data Liberation (PUDL) project. PUDL's data and documentation are published under CC-BY-4.0.
• —Weather data by [Open-Meteo.com](https://open-meteo.com/) (CC BY 4.0). ERA5: Hersbach et al. (2023), ECMWF, doi:10.24381/cds.adbb2d47.
• —Population estimates: U.S. Census Bureau (US public domain).

Built and maintained by **Arimancy** (Arimancy LLC) — dataset page: arimancy.com/datasets/ercot-hourly-demand-weather. Also available on Kaggle and Hugging Face. Column-by-column reference: the included data_dictionary.md.