deusmos/cbb26-day-corpus-v1

CBB26 Day Corpus v1 (cbb26-day-corpus-v1)

Materialized `day_corpus_bundle_v1` training corpora derived from cbb26 Timescale replay — published for cloud training and research without re-running materialization locally.

Canonical Hub repo: `deusmos/cbb26-day-corpus-v1`

Source-of-truth for this card (edit here, then publish): docs/datasets/cbb26-day-corpus-v1/README.md in the cbb26 git repo.

Standards index: `docs/standards/README.md`

Start here

Not a row-oriented HF dataset. Files are NPZ/JSON day bundles. Do not expect `load_dataset()` to return training rows. Use hf download + verify scripts below.

NDDS = Normalized Decimated Day Slabs — 10-second decimated, loader-normalized tensor slabs cached under _ndds_cache/ (~300 MiB/day on Hub). Implementation: `libs/day_raster_corpus/ndds_cache.py`.

What this dataset is

This is not raw Timescale replay. It is the materialized corpus layer above replay — one fixed materialization profile (10-second decimation, loader normalization, 128-row windows per day_corpus_bundle_v1). That profile is lossy by design: it encodes cbb26's training geometry, not every choice you might want for your own experiments.

If you need a different bucket size, normalization policy, or decimation rate, start from `cbb26-timeseries-db` replay shards and materialize locally. This dataset is the convenience layer; replay preserves your options.

Hub layout (NDDS-only default)

train/{YYYY-MM-DD}/
  bundle_manifest.json
  anchors.npz
  _ndds_cache/metadata.json
  _ndds_cache/decimated_slabs.npz
eval/{YYYY-MM-DD}/…

~315 MiB/day average (NDDS-only). Full tensor_*.npz files are not on Hub by default — materialize locally from replay if you need raw slabs.

Schema

• —Manifest: schema_version: day_corpus_bundle_v1
• —Spec: `docs/standards/specs/DAY_CORPUS_BUNDLE_v1.md`

Coverage snapshot (live)

<!-- COVERAGE:START -->

Refresh: uv run python scripts/generate_hf_dataset_coverage.py --update-readme

<!-- COVERAGE:END -->

Download

Quick verify path (~10–15 min):

git clone https://github.com/deusmos/cbb26.git && cd cbb26
uv sync
./examples/download_and_verify_corpus_day.sh

Manual download:

export HF_TOKEN=...   # optional for public repo
export HF_DATASET_REVISION=<pin-sha-after-upload>

uv run hf download deusmos/cbb26-day-corpus-v1 \
  --repo-type dataset \
  --revision "$HF_DATASET_REVISION" \
  --local-dir ./corpus-hf

Single day:

uv run hf download deusmos/cbb26-day-corpus-v1 \
  --repo-type dataset \
  --include "train/2024-12-01/*" \
  --local-dir ./corpus-hf

Verify locally

uv run python scripts/verify_day_raster_bundle_dir.py \
  ./corpus-hf/train/2024-12-01 --require-ndds

Root verify (profile days):

uv run python scripts/verify_free_17d_day_raster_output_root.py ./corpus-hf/train

Tools for researchers

Visual preview (local)

One command after clone — download one train day, render window PNGs, open HTML gallery:

./examples/preview_corpus_day.sh
# opens file://.../tmp/examples_corpus_previews/index.html

Underlying render script: scripts/render_day_bundle_window_samples.py (writes gallery-compatible *_summary.json sidecars).

Visual previews (Hub)

Static montage PNGs are published under previews/{split}/{YYYY-MM-DD}/ with an index at `previews/manifest.json`.

Colormap: signed quantity / depth / delta channels mapped to RGB via `libs/tensor_materializer/visualization.py` (tensor_to_rgb_image_array).

Maintainers regenerate thumbnails:

scripts/upload_corpus_previews_to_hf.sh --corpus-root ./corpus-hf

PyTorch DataLoader (research)

Use the shareable loader module (wraps verified NDDS window code; no latent-encoder labels):

from libs.day_raster_corpus.pytorch_loader import Cbb26DayCorpusDataset, load_day_corpus_dataloader

# After: ./examples/download_and_verify_corpus_day.sh
day_dir = "./tmp/examples_corpus_day/train/2024-12-01"

dataset = Cbb26DayCorpusDataset(day_dir, ndds_cache_mode="readonly")
sample = dataset[0]
# sample["tensor"]: (P, 128, bucket_count, 3) float16
# sample["anchor_row_decimated"], sample["anchor_second_index"], sample["trade_date"]

loader = load_day_corpus_dataloader(day_dir, batch_size=4, shuffle=True, ndds_cache_mode="readonly")
batch = next(iter(loader))
# batch["tensor"]: (B, P, 128, bucket_count, 3)

Multi-day local corpus:

loader = load_day_corpus_dataloader("./corpus-hf", split="train", batch_size=8)

Stream from Hub without a full local tree (requires HF_TOKEN if gated):

dataset = Cbb26DayCorpusDataset(
    "train/2024-12-01",
    corpus_source="hf",
    hf_dataset_repo="deusmos/cbb26-day-corpus-v1",
    ndds_cache_mode="readonly",
)

Notes:

• —Prefer num_workers=0 — full-day NDDS slabs are held in process memory.
• —Default normalization: sample_window_max_abs_per_channel (see libs/day_raster_corpus/loader_normalization.py).
• —Raw timeseries is not a PyTorch loader. Restore `cbb26-timeseries-db` to Postgres and materialize day bundles first.

Demo script: `examples/pytorch_day_corpus_dataloader.py`

Produce bundles from replay

Requires populated Timescale (collector, restore, or backfill):

DB_HOST=localhost DB_PORT=5432 \
uv run python scripts/materialize_valid_day_bundles.py \
  --output-root ./corpus \
  --dates 2024-12-01

See Quickstart Path B and timeseries dataset card.

Upload (maintainers)

export HF_DATASET_REPO=deusmos/cbb26-day-corpus-v1
scripts/upload_corpus_to_hf_sequential.sh \
  --corpus-root ./corpus \
  --batch-by day

Default: --corpus-artifacts ndds-only. Full tensors: --corpus-artifacts full.

Publish this README:

scripts/publish_day_corpus_dataset_readme.sh

Publish Hub preview thumbnails (maintainers):

scripts/upload_corpus_previews_to_hf.sh --corpus-root ./corpus-hf

Contributing missing days

1. 1.Materialize and verify locally (verify_day_raster_bundle_dir.py)
2. 2.Upload via sequential script or open Hub Discussion for promotion
3. 3.See `docs/standards/CONTRIBUTING_DATA.md`

FAQ

Can I use this commercially?

Market data is governed by Coinbase market data terms. Software and manifests are MIT licensed. Not legal advice.

Why Postgres dumps not Parquet?

Raw replay uses pg_dump; this corpus layer uses NPZ/JSON bundles. See ADR-002 and ADR-003.

Why two datasets?

Raw replay (timeseries-db) vs materialized NDDS (this repo).

How complete is UTC day X?

Bundle manifest lists products; upstream replay day should have 20/20 shards before materialization.

How do I cite this?

@dataset{cbb26_day_corpus_v1,
  title  = {CBB26 Day Corpus v1 (NDDS Day Bundles)},
  author = {deusmos},
  year   = {2026},
  url    = {https://huggingface.co/datasets/deusmos/cbb26-day-corpus-v1}
}

Is this affiliated with Coinbase?

No. Independent research project using public market feeds.

Full FAQ source: `docs/datasets/_shared/FAQ.md`.

Licensing and data use

• —Software / manifests: MIT License
• —Market data: derived from Coinbase feeds — see Coinbase market data terms and `DATA_USE_POLICY.md`

References

Changelog