TSFM-ai/future-ts

FUTURE-TS

Summary

FUTURE-TS v0.1.0 is a public-preview benchmark for time-series foundation models. It treats evaluation as an executable protocol: task cards declare issue times, horizons, available history, delayed labels, adaptation budgets, resource limits, and metrics; submissions are validated, scored, and audited against those contracts.

The core rule is future-only ranking: a submission is only ranked on labels that were not available when the model was frozen.

Benchmark Surface

• —Release: v0.1.0
• —Strict surface: benchmarks/v1/benchmark.json
• —Surface identifier: future-ts-v1
• —Tasks: 25
• —Tiers: public_dev, blind_archive, live
• —Tracks: forecasting core, covariate-aware, multivariate relational, data quality, event, transfer, multimodal context
• —Required manifest: require_pretraining_manifest=true
• —Headline score: tier_weighted_score
• —Additional views: capability vector (F, U, R, A, E, D), Pareto flag, rank-based mean-of-ranks aggregate with bootstrap CI

The strict surface rejects manifestless submissions so "clean" and "undeclared" are not conflated.

Current Empirical Run

The current canonical empirical artifacts are in:

reports/tsfm_ai_empirical_v2_multi_budget/

This run is the future-ts-empirical-v2 hosted TSFM.ai slice used by the empirical paper. It exercises 15 real-data tasks across three context budgets:

Current result snapshot:

• —Surfaced public catalog entries: 52
• —Merged public submissions: 51
• —Scored public models: 47
• —Positive overall scores: 30
• —Current winner: Datadog/Toto-2.0-1B
• —Winner score: 0.2369
• —Rank-consistency leader by mean rank: amazon/chronos-2

Top five by tier-weighted score:

How To Submit A Model Entry

Submit a pull request under:

submissions/community/<org>_<model>/

Each submission directory must contain:

• —script.py: sealed-runner entry point
• —declaration.json: metadata, declarations, artifact URI, and pretraining manifest
• —README.md: short model description and links

CI validates the directory structure and smoke-runs script.py through the sealed runner. After review, the evaluator runs the full benchmark and can publish the resulting BenchmarkReport.

See submission-guide.md for the full process.

What This Does Not Yet Claim

The v0.1.0 release is a local benchmark package plus sealed-runner MVP. It is not yet a fully hosted, externally attested live benchmark service.

Do not read the current empirical run as a permanent universal TSFM ranking. It is a first real-data slice over 15 tasks and one materialized wave. Wider task coverage, repeated waves, hosted attestation, immutable submission windows, and stronger manifest evidence are the next steps.

See validity-envelope.md for the precise claim boundary.

Local Commands

Validate the strict benchmark:

python3 -m future_ts.cli validate-benchmark benchmarks/v1/benchmark.json

Run the sealed reference submission:

python3 -m future_ts.cli run-sealed \
  examples/submissions/reference_seasonal_naive.py \
  .tmp/task_windows.json \
  --output .tmp/submission.json \
  --submission-id local::reference_seasonal_naive \
  --model-name "Reference Seasonal Naive"

Build a leaderboard from saved reports:

python3 -m future_ts.cli leaderboard reports/tsfm_ai_empirical_v2_multi_budget/*.report.json

These core workflows do not require TSFM.ai. A TSFM.ai API key is only needed to run the optional hosted-catalog evaluation commands that invoke hosted models.

More Detail

• —Design paper: paper/future_ts_design.pdf
• —Empirical paper: paper/future_ts_empirical.pdf
• —Benchmark design notes: benchmark-design.md
• —Submission guide: submission-guide.md
• —Sealed runner: sealed-runner.md
• —Validity envelope: validity-envelope.md

Repository Contents

• —benchmarks/: public benchmark surfaces and task-card references.
• —schemas/: JSON Schemas for benchmark, task, submission, and actual records.
• —docs/: benchmark card, submission guide, sealed-runner notes, and validity envelope.
• —reports/tsfm_ai_empirical_v2_multi_budget/: canonical public empirical reports.
• —paper/: design and empirical PDFs.