CoolFace
Datasetpublic

TommyKwok/institutional-core-2024-q1

Institutional Core 2024 Q1 — Market Data Product Production-grade, backtest-ready market data for 12 core asset proxies spanning FX, Equities, Crypto, and Commodities. Built on the Universal Market Data Platform with full lineage tracing, quality certification, and reproducibility receipts. Dataset Details Property Value Period 2024-01-01 → 2024-01-14 Base timeframe m1 (1-minute candles) Total rows 238,260 Certified ✅ is_certified=True… See the full description on the dataset page: https://huggingface.co/datasets/TommyKwok/institutional-core-2024-q1.

sourceHugging Facemitupdated 4mo agoView on Hugging Face
1likes33downloads
Dataset Card

Institutional Core 2024 Q1 — Market Data Product

Production-grade, backtest-ready market data for 12 core asset proxies spanning FX, Equities, Crypto, and Commodities. Built on the Universal Market Data Platform with full lineage tracing, quality certification, and reproducibility receipts.

Dataset Details

PropertyValue
Period2024-01-01 → 2024-01-14
Base timeframem1 (1-minute candles)
Total rows238,260
Certified✅ is_certified=True
Completeness1.0 (all symbols)
DuckLake snapshot@1665
Target variablefuture_return (horizon=10 bars)
SourceDukascopy datafeed
Versionv9 (semantic: 9.0.0)

Asset Coverage

Asset ClassSymbolRows (m1)
FXDUKASCOPY:EURUSD20,100
FXDUKASCOPY:GBPUSD20,100
FXDUKASCOPY:USDJPY20,100
Equity IndexDUKASCOPY:USA500IDXUSD (ES proxy)20,100
Equity IndexDUKASCOPY:USATECHIDXUSD (NQ proxy)20,100
Equity IndexDUKASCOPY:USA30IDXUSD (DJI proxy)20,100
CryptoDUKASCOPY:BTCUSD20,100
CryptoDUKASCOPY:ETHUSD20,100
CryptoDUKASCOPY:LTCUSD17,280
CommodityDUKASCOPY:XAUUSD (Gold)20,100
CommodityDUKASCOPY:XAGUSD (Silver)20,100
CommodityDUKASCOPY:LIGHTCMDUSD (Crude Oil)20,100

Schema

Each row contains:

  • —timestamp — UTC datetime (minute-level)
  • —symbol — USF format (VENUE:INSTRUMENT)
  • —open, high, low, close — OHLC prices (float)
  • —open_fixed, high_fixed, low_fixed, close_fixed — 1e-9 fixed-point integers (institutional precision)
  • —volume — traded volume
  • —venue, canonical_id, instrument_id — lineage identifiers
  • —timeframe — m1
  • —is_certified, is_healthy — quality flags
  • —target — future_return (10-bar horizon, computed without look-ahead bias)
  • —Feature columns (momentum, volatility) joined from Gold feature layer

Gold Layer (Resampled)

Additional resampled candles are available in the warehouse:

  • —m5: 3,685 rows/symbol
  • —h1: 335 rows/symbol

Lineage

Inlets:
  ducklake://silver/candles@1665   ← Dukascopy datafeed
  ducklake://gold/features@1665    ← Feature engine v3.1.0

Pipeline

Built with the Universal Market Data Platform:

  • —Extraction: DLT + Dukascopy Node CLI (monthly chunking)
  • —Transformation: Polars LazyFrame (Bronze → Silver → Gold Medallion)
  • —Orchestration: Prefect 3 composable flows
  • —Storage: DuckLake (SQLite catalog + Parquet data)
  • —Quality: Pandera schema validation + Readiness Audit

Usage

python
import polars as pl

df = pl.read_parquet("data/v9/materialized_institutional_core_2024_q1_v9/*.parquet")
print(df.shape)  # (238260, N)
print(df.group_by("symbol").agg(pl.len()).sort("symbol"))

Reproduction

bash
git clone https://github.com/Leo4815162342/dukascopy-node
cd dukascopy-node/examples/python-pipeline
uv run python scripts/build_q1_2024_production.py

License

MIT — free for research and commercial use with attribution.