Saxon0520/LiveHouse-TS-test

LiveHouse-TS

LiveHouse-TS is a prospective benchmark for univariate time-series forecasting. Models receive only observations that were publicly available at the forecast cutoff. Their predictions are frozen before the target window begins and scored after the complete target becomes available.

Website: https://huggingface.co/spaces/Saxon0520/LiveHouse-TS-test · Public data: https://huggingface.co/datasets/Saxon0520/LiveHouse-TS-test · Source: https://github.com/ATMSaxon/LiveHouse-TS-test

Core workflow

public sources -> normalized observations -> frozen forecast
               -> future observations arrive -> metrics -> public leaderboard

The public SDK has five modules:

• —data.py: streaming-data adapters and SQLite ingestion;
• —data_schema.py: canonical objects and schema v1;
• —models.py: the model protocol, HTTP client, and one baseline;
• —metrics.py: versioned point and probabilistic metrics;
• —eval.py: forecast freezing, scoring, and public export.

Runtime data, model weights, raw responses, frozen forecasts, and result files are not stored in this Git repository.

Quick start

git clone https://github.com/ATMSaxon/LiveHouse-TS-test.git
cd LiveHouse-TS
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'
pytest -q

Create or open a schema-v1 database:

from livehouse_ts.data_schema import connect

database = connect("livehouse.sqlite")

Implement DataSource.fetch() to return a normalized DataBatch, then call collect(source, SQLiteRepository(database)). A data adapter is responsible for recording event time, public availability time, and ingestion time separately.

Run one real cycle against the built-in Shanghai temperature and PM2.5 feeds:

livehouse-ts-cycle private/livehouse.sqlite public

This collects past observations, resolves due tasks, issues the next six-hour forecasts, and writes public CSV/JSON artifacts. Seasonal Naive is always included. Add HTTPS models through LIVEHOUSE_MODELS_JSON:

[{"model_id":"organization/model","endpoint_url":"https://forecast.example.org/forecast"}]

Join the benchmark

Paper model roster

LIVEHOUSE_MODEL_SET=baselines enables Seasonal Naive, Moving Average (24), ARIMA(1,1,1), and ETS (additive trend, no seasonality). LIVEHOUSE_MODEL_SET=paper additionally enables all eight foundation models:

Install .[baselines] for statistical inference and additionally .[tabpfn] for the paper roster. Core imports remain dependency-light. The TabPFN package versions match the previous repository's client environment; its hosted checkpoint is provider-controlled, so this does not establish exact reproduction of the paper's historical weights. The other hosted model IDs are also not immutable weight hashes.

In Actions, set secrets TSFM_API_KEY and TABPFN_TOKEN, then manually run Validate paper models with paper. This makes real provider calls on a synthetic series, uses inference quota, and does not publish benchmark results. Only after it passes, set repository variable LIVEHOUSE_MODEL_SET=paper to enable the hourly operator. minimal is the default and keeps the reference baseline plus explicitly configured HTTPS endpoints. Missing credentials fail before an operator run can change HF data.

These are new schema-v1 runs, not imported paper scores. Statistical models use a deterministic 200-sample centered residual bootstrap; failed fits are recorded as failures, never silently replaced by a different model. TiRex asks for at least 24 future steps and retains the requested prefix, matching the old registry's hosted-horizon workaround. The persistence demo is not a paper model.

Submit your own model

Participants host their model behind one small HTTP API. Start from `examples/model_api`, replace forecast_one, and expose:

GET  /health
POST /forecast

The request contains opaque series IDs, historical timestamps and values, frequency, horizon, and requested quantiles. It never contains future targets, private metrics, dataset-internal IDs, or another model's predictions. Remote endpoints must use stable HTTPS.

Each input must have one output with prediction_length finite numeric values in mean and in each requested quantile (normally 0.1, 0.5, 0.9). Quantiles must be nondecreasing at each time step. Point-only models may repeat their point forecast at every quantile, as the demo does; this is a degenerate distribution, not a calibrated uncertainty estimate. Admission and live evaluation apply the same checks before storing predictions.

Validate an endpoint before submission:

livehouse-ts-validate organization/model https://forecast.example.org/forecast

Submit the following through the GitHub community-model form:

1. 1.model ID and display name;
2. 2.model card URL;
3. 3.public endpoint-code URL;
4. 4.stable HTTPS /forecast URL;
5. 5.version and organization.

After endpoint validation, maintainers record an admission timestamp. A model is evaluated only on tasks created after admission; no historical backfill is used.

Evaluation protocol

For each task, LiveHouse-TS:

1. 1.selects context whose available_time is no later than the task cutoff;
2. 2.calls every admitted model with the same context and horizon;
3. 3.freezes predictions before target_start;
4. 4.waits until the target window ends and every target value is available;
5. 5.computes metrics with a recorded metric version;
6. 6.publishes only schema-v1 aggregate results and bounded visual examples.

Old pre-schema results are intentionally excluded from the new leaderboard.

Metrics and ranking

The canonical point metrics are MSE, RMSE, and MAE. MAPE is diagnostic only and is omitted when targets are close to zero. Probabilistic forecasts are evaluated with CRPS approximated from the common quantile grid.

MSE and CRPS are normalized to Seasonal Naive on the same task, averaged within each dataset, then averaged equally across datasets. Their mean is the primary score. The public table also reports average rank, pairwise win rate on shared releases, Elo, relative-to-naive gain (RTG), temporal stability, availability, failure count, and coverage. Models without enough releases remain visible but unranked. Implementations are versioned in metrics.py and eval.py.

Storage and deployment

The private Hugging Face Dataset contains the canonical SQLite database, normalized observations, tasks, and frozen forecasts. The public Dataset contains stable CSV, JSON, and metrics-only release exports. The Hugging Face Space is a read-only presentation layer with no token or private-data access. A single evaluator writer updates Saxon0520/LiveHouse-TS-test-private-data first, then publishes derived artifacts to the public Saxon0520/LiveHouse-TS-test Dataset using optimistic commit checks.

GitHub Actions provides an hourly single-writer operator and a manual HF resource/Space deployment workflow. Both use the repository secret HF_TOKEN. Optional external models are stored in LIVEHOUSE_MODELS_JSON. The Static Space itself receives no secret.

Run Verify Hugging Face backup in GitHub Actions to check recovery without changing either Dataset. It restores the private SQLite revision referenced by the public export, checks database integrity and foreign keys, and reproduces the four public CSV tables and leaderboard JSON (except its generation time). The run reports counts only; it does not upload the private database. The public Dataset exposes leaderboard, datasets, models, and releases as separate Dataset Viewer configurations.

Historical ranks and Backtesting Archive

The Space includes Elo and composite-score comparison charts, daily rank history, and a version selector that loads the table, datasets, and forecast example from the same pinned public revision. Elo is displayed separately; the primary rank continues to use the normalized composite score.

The existing hourly operator adds one snapshot per UTC day to history.json, using the previously published HF commit. The first captured version for a day is retained unchanged. Each archive entry links to that commit's leaderboard, release scores, forecast example, and private-source revision reference; it does not publish the private database. These are archived prospective results, not retrospective inference or imported paper scores. Archiving starts when this feature is deployed; outages leave gaps instead of fabricated history. The chart shows the latest 30 archived days; older versions remain selectable and downloadable. HF commit history must be retained for archive links to work.

LiveHouse series

This repository is intentionally scoped to univariate time series. Future spatio-temporal graph or multivariate benchmarks will use separate LiveHouse projects while sharing versioned concepts such as datasets, entities, variables, tasks, models, and releases.

Update log

0.1.0 — schema v1

• —Reduced the public project to five core modules and one HTTP model example.
• —Added canonical SQLite storage with separate event, availability and ingest times.
• —Added the livehouse-ts-v1 model and metric protocols.
• —Added two real Open-Meteo streams and the hourly single-writer operator.
• —Added normalized ranking, win rate, Elo, RTG, stability and availability.
• —Started a new leaderboard containing only schema-v1 releases.

Future schema, protocol, metric, data-source, or ranking changes receive an explicit version entry here.

Citation

@misc{livehouse_ts,
  title  = {LiveHouse-TS: A Live Benchmark for Time-Series Forecasting},
  author = {ThinkCat Lab},
  year   = {2026},
  url    = {https://github.com/ATMSaxon/LiveHouse-TS-test}
}

Apache-2.0 licensed. LiveHouse-TS builds on the prospective evaluation direction of GIFT-Eval while maintaining its own live-data schema and protocol.