KEDevO/crypto-market-datasets

Crypto Market Data Lake

Created and maintained by [Eimantas Kulbe](https://huggingface.co/KEDevO)

⭐ If you use this dataset in research, a product, or any publication, please cite the author (see Citation below). It took significant infrastructure and months of collection effort — a citation is the simplest way to give credit.

A continuously growing data lake of crypto market microstructure data sourced from Binance and alternative data providers. Full coverage across all 10 symbols — complete tick-level history from genesis to present, updated monthly.

Version `v202626` — BTC/ETH/SOL/BNB/XRP/DOGE full history, May 2026 snapshot.

All files are compressed Parquet (Snappy), sized 50–300 MB each for efficient streaming and DuckDB / pandas compatibility. All timestamps are int64 milliseconds UTC.

Symbols & Coverage

Datasets included

Note: BTC futures agg_trades Sep–Dec 2019 unavailable (Binance vault 404).

File layout

raw/
  agg_trades/
    market={spot,futures}/symbol={BTCUSDT,ETHUSDT,SOLUSDT}/year=YYYY/month=MM/
      {start_ms}_{end_ms}.parquet          ← 50–300 MB each
  klines/
    market={spot,futures}/symbol={BTCUSDT,ETHUSDT,SOLUSDT}/interval=1m/year=YYYY/month=MM/
      {start_ms}_{end_ms}.parquet
  funding_rates/symbol=BTCUSDT/year=YYYY/month=MM/
  open_interest/symbol=BTCUSDT/year=YYYY/month=MM/
  premium_index/symbol=BTCUSDT/year=YYYY/month=MM/
  alternative_me/fear_greed/year=YYYY/
  deribit/year=YYYY/month=MM/
  macro/{dxy,spx,gold,us10y}/year=YYYY/

Schemas

agg_trades

klines

funding_rates

Quick-start (DuckDB)

import duckdb

# Spot aggregate trades for 2021 — full year, fast scan
df = duckdb.sql("""
    SELECT date_trunc('hour', to_timestamp(timestamp/1000)) AS hour,
           sum(quantity * price) AS usd_volume,
           count(*)              AS trade_count
    FROM read_parquet(
        'raw/agg_trades/market=spot/symbol=BTCUSDT/year=2021/**/*.parquet',
        hive_partitioning=true
    )
    GROUP BY 1 ORDER BY 1
""").df()

# All-history 1-minute klines (spot)
klines = duckdb.read_parquet(
    'raw/klines/market=spot/symbol=BTCUSDT/interval=1m/**/*.parquet',
    hive_partitioning=True
)

Data sources

Collection methodology

Built and maintained by Eimantas Kulbe using a custom Python pipeline running on four VPS nodes with rotating HTTP proxy pools to handle Binance rate limits without API keys.

• —Pre-2022 aggregate trades: downloaded from the Binance Data Vault monthly ZIP archives; parsed with DuckDB for out-of-core efficiency
• —2022–2026 aggregate trades: collected via Binance REST API using 1-hour time-range windows, distributed across workers with modulo assignment
• —Klines / Funding rates / OI: Binance REST API, full history
• —Real-time data: Binance WebSocket public streams (no API key required)
• —Alt data: Deribit REST, alternative.me, Yahoo Finance — polled every 15 minutes to 1 hour

Infrastructure: all writes are idempotent with a PostgreSQL chunk-log tracking every hourly window (done | skipped | failed). Schema version 1 is embedded in every Parquet file's metadata for forward-compatibility.

Known gaps

License

This dataset is released under the [Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/) license.

You are free to share, adapt, and use this data for any purpose including commercial, as long as you give appropriate credit to the author.

⚠️ Attribution is required. If you use this dataset, you must cite Eimantas Kulbe as the author. See the Citation section below.

Citation

If you use this dataset in academic work, a commercial product, a blog post, or any other publication, please cite it. This is a large-scale data collection effort and proper attribution helps sustain open data projects.

BibTeX:

@dataset{kulbe_btcusdt_2026,
  author    = {Kulbe, Eimantas},
  title     = {Crypto Market Data Lake (Binance + Alt Data)},
  year      = {2026},
  publisher = {HuggingFace Datasets},
  url       = {https://huggingface.co/datasets/KEDevO/crypto-market-datasets},
  license   = {CC BY 4.0},
  note      = {Contains aggregate trades, OHLCV klines, funding rates,
               open interest, and alternative data for BTCUSDT, ETHUSDT,
               and SOLUSDT on Binance, through May 2026.}
}

APA:

Kulbe, E. (2026). Crypto Market Data Lake (Binance + Alt Data) [Data set]. HuggingFace Datasets. https://huggingface.co/datasets/KEDevO/crypto-market-datasets

Acknowledgement (for papers):

The cryptocurrency market data used in this work was sourced from the Crypto Market Data Lake compiled by Eimantas Kulbe (2026), available at https://huggingface.co/datasets/KEDevO/crypto-market-datasets under CC BY 4.0.

Dataset compiled and maintained by [Eimantas Kulbe](https://huggingface.co/KEDevO). For issues or questions, open a discussion on the dataset page.