deusmos/cbb26-timeseries-db

CBB26 Timescale market data (cbb26-timeseries-db)

Public Timescale/Postgres replay shards for the cbb26 monorepo: canonical Coinbase Advanced Trade level-2 order book history stored in the market_data schema, packaged as restorable pg_dump files for research, corpus materialization, and reproducibility.

Canonical Hub repo: `deusmos/cbb26-timeseries-db`

Source-of-truth for this card (edit here, then publish): docs/datasets/cbb26-timeseries-db/README.md in the cbb26 git repo.

Start here

Not a row-oriented HF dataset. Files are pg_dump shards and JSON sidecars. Do not expect `load_dataset()` to return tabular rows. Use hf download + the restore workflow below.

NDDS = Normalized Decimated Day Slabs — 10-second decimated, loader-normalized tensor slabs cached under _ndds_cache/ (~300 MiB/day on Hub). For pre-built NDDS bundles use `cbb26-day-corpus-v1`. Implementation: `libs/day_raster_corpus/ndds_cache.py`.

Why replay shards?

Timescale replay is base truth for what happened on the Coinbase L2 order book: checkpoints, anchors, and per-second deltas with carry-forward semantics. Materializing to tensors is lossy — bucket sizes, normalization, and decimation are design choices, not facts stored in the exchange feed.

Publishing replay as restorable pg_dump shards lets you pick your own materialization profile instead of being locked into the maintainer's. `cbb26-day-corpus-v1` is the fast path when you want our standard geometry; this repo is for when you need fidelity and flexibility. Restore + materialize takes longer — that is an intentional tradeoff.

What this dataset is

This is not a training corpus of pre-built tensors. It is the raw replay layer that cbb26 services write and read before materialization:

Schema (market_data)

Each shard exports the four core replay tables required for deterministic order-book replay in cbb26:

DDL and migrations ship in-repo on the dataset:

• —migrations/0001_init.sql … 0004_replay_metadata_window_end_index.sql
• —schema/market_data_schema.dump — schema-only pg_dump -Fc for empty restores

Hypertables use TimescaleDB 2.x on Postgres 16. See services/timeseries-db/migrations/ in cbb26 for the live definition.

Products (20)

Shards use Coinbase product ids (BASE-QUOTE):

AAVE-USD, ADA-USD, APT-USD, ATOM-USD, AVAX-USD, BCH-USD, BTC-USD, DOGE-USD, DOT-USD, ETH-USD, HBAR-USD, LINK-USD, LTC-USD, NEAR-USD, PEPE-USD, SHIB-USD, SOL-USD, UNI-USD, XLM-USD, XRP-USD

This matches the default `MARKET_COLLECTOR_PRODUCTS` basket in deploy/compose/slice-a.yml.

Coverage snapshot (live)

<!-- COVERAGE:START -->

Refresh: uv run python scripts/generate_hf_dataset_coverage.py --update-readme

<!-- COVERAGE:END -->

Each shard sidecar (data/.../{product}.json) records row_counts, dump_size_bytes, and UTC day bounds (cbb26_timeseries_shard_manifest_v1).

Data dictionary (replay rows)

Full contract: `docs/standards/specs/STORAGE_REPLAY.md` · JSON source: `libs/contracts/storage_and_snapshot_spec.json`

Example `orderbook_second_deltas` row (sanitized static illustration):

{
  "product_id": "BTC-USD",
  "changed_second": "2024-12-01T12:34:56+00:00",
  "source_sequence_num_start": 9876543210,
  "source_sequence_num_end": 9876543299,
  "best_bid": "96543.21000000",
  "best_ask": "96543.22000000",
  "changes": [
    ["bid", "96540.00", "1.25000000"],
    ["offer", "96543.22", "0.50000000"],
    ["offer", "96544.00", "0"]
  ],
  "change_count": 3
}

Repository layout

deusmos/cbb26-timeseries-db/
├── README.md                          ← dataset card (this file on Hub)
├── backup_manifest.json               ← export inventory + restore notes (updated at bootstrap)
├── schema/
│   └── market_data_schema.dump        ← schema-only pg_dump (market_data)
├── migrations/
│   ├── 0001_init.sql
│   ├── 0002_checkpoint_delta_upgrade.sql
│   ├── 0003_replay_anchor_support.sql
│   └── 0004_replay_metadata_window_end_index.sql
└── data/
    └── {YYYY-MM-DD}/
        ├── {PRODUCT}.dump             ← pg_dump -Fc (core-replay tables for that day+product)
        └── {PRODUCT}.json             ← shard sidecar (row counts, sizes)

Naming conventions

• —`{YYYY-MM-DD}` — UTC calendar trade date (inclusive day window 00:00:00 … 23:59:59 UTC).
• —`{PRODUCT}` — uppercase Coinbase id, e.g. BTC-USD (regex ^[A-Z0-9]+-[A-Z0-9]+$).
• —One Hub commit per shard during sync upload (rate-limit friendly; ~30 s spacing).

Manifests and revision pinning

There are no separate checksum files; verify restores with sidecar row_counts, pg_restore exit code, and scripts/smoke_restore_hf_shard.sh.

Download and restore

Quick inspect (no Postgres)

git clone https://github.com/deusmos/cbb26.git && cd cbb26
./examples/download_one_timeseries_shard.sh

Prerequisites

• —Docker (for pg_dump/pg_restore client Postgres 16 — use image pgvector/pgvector:pg16 or timescale/timescaledb:2.14.2-pg16)
• —A running Postgres 16 + TimescaleDB target (see Run your own collector below)
• —Hugging Face CLI: uv run hf auth login or HF_TOKEN in environment

Download

Full dataset (large):

export HF_TOKEN=...   # optional for public repo; required for uploads
uv run hf download deusmos/cbb26-timeseries-db --repo-type dataset --local-dir ./cbb26-timeseries-db

Pin a revision (recommended for reproducible materialization):

export HF_TIMESERIES_DATASET_REVISION=b6c9773744ca346dba5c37c05939349f1ae3225e  # example; get latest from Hub
uv run hf download deusmos/cbb26-timeseries-db \
  --repo-type dataset \
  --revision "$HF_TIMESERIES_DATASET_REVISION" \
  --local-dir ./cbb26-timeseries-db

Single day / product:

uv run hf download deusmos/cbb26-timeseries-db \
  --repo-type dataset \
  --include "data/2024-12-01/BTC-USD.*" \
  --local-dir ./cbb26-timeseries-db

Python (huggingface_hub):

import os

from huggingface_hub import snapshot_download

path = snapshot_download(
    repo_id="deusmos/cbb26-timeseries-db",
    repo_type="dataset",
    revision=os.environ["HF_TIMESERIES_DATASET_REVISION"],  # pin SHA
    allow_patterns=["data/2024-12-01/*", "migrations/*", "schema/*"],
)

Restore workflow

1. 1.Start Timescale with an empty or compatible database (see collector section).
2. 2.Apply migrations if the volume is fresh:

   # migrations are in the downloaded tree under migrations/
   psql -h 127.0.0.1 -U cbb26 -d cbb26 -f migrations/0001_init.sql
   # … or rely on docker-entrypoint-initdb.d on first volume init

1. 1.Restore schema (if needed):

   docker run --rm --network host \
     -e PGPASSWORD="$POSTGRES_PASSWORD" \
     -v "$PWD/cbb26-timeseries-db:/in" \
     pgvector/pgvector:pg16 \
     pg_restore -h 127.0.0.1 -p 5432 -U cbb26 -d cbb26 --schema-only \
     /in/schema/market_data_schema.dump

1. 1.Restore one shard (data-only into staging schema):

   scripts/smoke_restore_hf_shard.sh \
     --shard-dir ./cbb26-timeseries-db \
     --date 2024-12-01 --product BTC-USD

Or manually with pg_restore + merge SQL (see script for merge statements).

1. 1.Verify sidecar expectations — script fails closed on row-count mismatch vs sidecar.

Replay invariant: Each shard includes the baseline replay anchor at or before day_start_utc. Without it, same-day replay cannot be reconstructed.

Run your own collector (local Docker stack)

You can collect the same schema locally and later upload missing shards (see Contributing).

Prerequisites

• —Docker + Docker Compose
• —Linux or macOS host with ~50+ GiB free for sustained multi-product collection
• —Network access to Coinbase Advanced Trade websocket

1. Configure environment

From the cbb26 repo root:

cp .env.example .env
# Edit POSTGRES_PASSWORD, optional ports, and MARKET_COLLECTOR_PRODUCTS

Key variables (see .env.example and deploy/compose/slice-a.yml):

Inside Compose containers, services use DB_HOST=timeseries-db (not localhost).

2. Start the stack

scripts/up_slice_a.sh
scripts/check_slice_a.sh

Services: timeseries-db, market-collector, tensor-materializer, prometheus, grafana.

3. Verify health

Expected endpoints:

• —Collector: http://localhost:8080/healthz, http://localhost:8080/readyz
• —Materializer: http://localhost:8081/healthz

After the collector runs, confirm replay rows exist:

DB_HOST=localhost DB_PORT=5432 uv run python scripts/_timeseries_db_hf_backup.py inventory --scope core-replay

Internal runbook: `docs/runbooks/slice-a.md`.

Contributing missing shards and days

See `docs/standards/CONTRIBUTING_DATA.md` (operator guide).

The dataset grows by incremental sync from contributors with a populated Timescale instance. Uploads are idempotent: shards already on Hub are skipped.

export HF_TIMESERIES_DATASET_REPO=deusmos/cbb26-timeseries-db
scripts/upload_timeseries_db_to_hf.sh --sync \
  --start-date 2026-05-22 --end-date 2026-05-31

Tools for researchers

This repo is replay truth, not pre-rendered tensors. To see what NDDS windows look like without restoring Postgres, use the day-corpus tooling:

Raw shard preview without Postgres restore is not shipped in v1 — restore + materialize (Path B) or use day-corpus previews. See shared FAQ.

FAQ

Can I use this commercially?

Market data is governed by Coinbase market data terms. Software and schema in cbb26 are MIT licensed. Not legal advice.

Why Postgres dumps not Parquet?

Replay fidelity requires full L2 book state. See ADR-003.

Why two datasets?

Raw replay (this repo) vs materialized NDDS day bundles (`cbb26-day-corpus-v1`).

How complete is UTC day X?

20/20 products under data/{YYYY-MM-DD}/; sidecar orderbook_second_deltas ≈ 86400 for a full continuous day.

How do I cite this?

@dataset{cbb26_timeseries_db,
  title  = {CBB26 Coinbase L2 Order Book Replay Shards},
  author = {deusmos},
  year   = {2026},
  url    = {https://huggingface.co/datasets/deusmos/cbb26-timeseries-db}
}

Is this affiliated with Coinbase?

No. Independent research project using public market feeds.

Full FAQ source: `docs/datasets/_shared/FAQ.md`.

Licensing and data use

• —Software / schema / scripts: MIT License (cbb26 repository).
• —Market data: sourced from Coinbase public market feeds. Respect Coinbase market data terms. Provided for research and reproducibility without warranty.

Policy: `docs/standards/DATA_USE_POLICY.md`.

Related (downstream)

Restoring HF shards into local Timescale lets you materialize new days/products without operator infrastructure. Pre-built NDDS day bundles for cloud training live in `deusmos/cbb26-day-corpus-v1`.

flowchart LR
  CB[Coinbase L2 websocket] --> MC[market-collector]
  MC --> TS[(timeseries-db / market_data)]
  TS --> TM[tensor-materializer / materialize scripts]
  TM --> CORP[Corpus artifacts]
  TS --> HF[(HF timeseries shards)]
  CORP --> HFC[(HF day-corpus-v1 NDDS)]
  CORP --> TR[train_latent_encoder.py]

Materialization entrypoints: scripts/materialize_valid_day_bundles.py, scripts/materialize_free_tardis_17d_day_bundles.py. Training: scripts/train_latent_encoder.py (v2 contract). Vast/JOJAT runbook: `docs/runbooks/vast_jojat_training.md`.

References

Changelog