tmmycruise/autoresearch-crypto-data

Binance public crypto market data

This public dataset contains typed, compressed, and audited copies of market data published for free by Binance at <https://data.binance.vision/>. It is organized for reproducible point-in-time research across the markets represented in the coverage report. The production backfill stores the complete compact aggregate layer for a pinned liquid USD-M and COIN-M futures universe, plus every option index and option-surface underlying published in the archive. Native trade and order-book event history is intentionally exhaustive only for BTC. The reader and schema also support spot data, but the presence of a spot example does not imply exhaustive spot-market coverage.

The archive is generated by the open-source pipeline in <https://github.com/MarketFuckery/Autoresearch>. It contains public market data only. It does not contain account, order, credential, or other private data.

Coverage and verification

The machine-readable coverage report is:

crypto/binance/v1/metadata/coverage.json

Use that report as the authority for completed markets, datasets, tickers, source objects, rows, bytes, and remote verification. Do not infer complete coverage from the existence of one file or from a calendar filename. Binance does not publish every dataset for every ticker and period.

During an active archive build, the report is published with finished: false. Treat its generated_at timestamp and missing-group lists as a progress snapshot. Only a report with finished: true, scope.verified: true, compaction.verified: true, and remote.verified: true proves final completion.

The pinned universe was selected on 2026-08-20 and contains 19 USD-M perpetual tickers with at least $50 million median daily notional, 10 COIN-M perpetual tickers with at least $1 million, both published BVOL indexes, and five published EOH option-surface underlyings. In this repository, "complete" means every available source object from 2020-01-01 through 2026-07-31 for that pinned universe and the dataset selection recorded on each universe row. all is used for BTC futures, core for other futures, and the named published dataset for option rows. It does not mean every Binance listing. The exact universe is versioned in the pipeline as config/binance_liquid_universe.csv.

For this pinned universe, final scope is 190 market/dataset/ticker groups:

• —184 aggregate or sampled-snapshot groups that must each resolve to one canonical all-history Parquet file; and
• —6 BTC native-event groups stored as complete, non-overlapping monthly Parquet partitions.

The compact aggregate foundation is built first for the entire pinned universe. The runner then finishes BTC aggregate trades, raw trades, best-bid/ask updates, sampled depth, and COIN-M liquidation data. Other futures tickers stop at the compact core: traded one-minute OHLCV, mark/index/premium one-minute prices, funding, and positioning metrics. Event files uploaded for another asset during an earlier build are opportunistic extras and do not imply complete event coverage; the machine-readable coverage report remains authoritative.

The final verifier checks that:

• —every inventoried Binance source object belongs to a terminal unit;
• —no phase state overlaps another phase state;
• —all required Parquet objects exist remotely at their audited byte size and LFS SHA-256;
• —compacted groups preserve source row counts and schema; and
• —superseded construction shards are absent after compaction.

Final verification also fails if any price, funding, positioning, BVOL, option surface, depth-snapshot, or liquidation-snapshot group does not resolve to exactly one canonical all_history/data.parquet path. Large event datasets are checked as complete, non-overlapping monthly partitions instead.

Each data object has a sibling source_manifest.json containing source keys, source SHA-256 hashes, row counts, time bounds, and the output audit. Parquet files use Zstandard compression.

Layout

The canonical root is:

crypto/binance/v1/

Construction shards are organized as:

market={MARKET}/dataset={DATASET}/ticker={TICKER}/year={YYYY}/period={PERIOD}/data.parquet
market={MARKET}/dataset={DATASET}/ticker={TICKER}/year={YYYY}/period={PERIOD}/source_manifest.json

Compact aggregate datasets are organized as one all-history file per market, dataset, and ticker:

market={MARKET}/dataset={DATASET}/ticker={TICKER}/all_history/data.parquet
market={MARKET}/dataset={DATASET}/ticker={TICKER}/all_history/source_manifest.json

How to select files

For one (market, dataset, ticker) group:

1. 1.If all_history/data.parquet exists, read that file only.
2. 2.Otherwise, read every year=*/period=*/data.parquet construction shard for the group.
3. 3.Never union all_history with the corresponding period shards. They represent the same logical rows.
4. 4.Pin a Hugging Face revision for any published experiment.

Small aggregate series are compacted into one all-history file after their backfill is complete and remotely verified. These include minute OHLCV, mark, index, and premium prices, funding, positioning metrics, option aggregates, BVOL, sampled depth, and liquidation histories.

Very large event streams remain monthly because a single file can become impractical to download, rewrite, or recover. These include individual trades, aggregate trades, and top-of-book events. Monthly event files are still one ticker and one dataset per file, with stable schemas and non-overlapping source periods.

The year and period components are storage partitions, not sampling frequencies. aggTrades, trades, and bookTicker retain Binance's native event rows and are not resampled to one minute. Likewise, an interval entry in Parquet schema metadata is pipeline configuration metadata and must not be used to infer event frequency. Use the dataset's primary timestamp columns and the actual row sequence instead.

Market identifiers

Ticker meaning depends on the market. BTCUSDT in spot and um is not the same instrument. COIN-M tradable contracts such as BTCUSD_PERP are distinct from the shared index pair BTCUSD. Always join on both market and symbol unless a documented cross-market mapping is intended.

Dataset catalog

Price and aggregate datasets

Event datasets

Canonical schemas and units

Every Parquet file includes a symbol column even though the path is already partitioned by ticker. Time columns sourced as integers are preserved twice:

• —{name}_raw contains Binance's original integer timestamp; and
• —{name} is normalized to timestamp[ns, tz=UTC].

This is important because Binance spot archives changed timestamp precision from milliseconds to microseconds on 2025-01-01.

Klines

Spot and USD-M kline columns are:

symbol, open_time_raw, open_time, open, high, low, close,
base_asset_volume, close_time_raw, close_time, quote_asset_volume,
trade_count, taker_buy_base_asset_volume,
taker_buy_quote_asset_volume, ignore

COIN-M is inverse-contract data. Its volume columns are named explicitly:

contract_volume, base_asset_volume,
taker_buy_contract_volume, taker_buy_base_asset_volume

Do not compare COIN-M contract counts directly with spot or USD-M base-asset volume. Use contract specifications and price to normalize notional.

Trades

Aggregate trades contain:

symbol, aggregate_trade_id, price, quantity,
first_trade_id, last_trade_id,
event_time_raw, event_time,
buyer_is_maker, best_price_match

If buyer_is_maker = true, the buyer was passive and the seller was the aggressor. If it is false, the buyer was the aggressor.

Individual trades rows contain:

symbol, trade_id, price, quantity,
quote_asset_quantity or base_asset_quantity,
event_time_raw, event_time,
buyer_is_maker, best_price_match

USD-M uses quote_asset_quantity; COIN-M uses base_asset_quantity. The main quantity field is base-asset quantity for USD-M and contract quantity for COIN-M. Do not add trades and aggTrades volumes together because aggregate trades summarize the same underlying executions.

Funding and BVOL

Funding rows contain:

symbol, calculation_time_raw, calculation_time,
funding_interval_hours, last_funding_rate

last_funding_rate is a dimensionless decimal rate for the stated interval. A value of 0.0001 is 0.01 percent for that interval. Do not assume an eight-hour interval when funding_interval_hours says otherwise.

BVOL rows contain:

symbol, calculation_time_raw, calculation_time,
source_symbol, base_asset, quote_asset, index_value

Treat index_value as Binance BVOL index points. Do not silently reinterpret it as a decimal implied volatility or mix it with option mark_iv without applying an explicitly documented scale conversion.

Top of book

bookTicker rows contain:

symbol, update_id,
best_bid_price, best_bid_quantity,
best_ask_price, best_ask_quantity,
transaction_time_raw, transaction_time,
event_time_raw, event_time

The bid/ask quantities follow the contract's market convention: base-asset quantity for USD-M and contract quantity for COIN-M. Update IDs can be used to detect gaps, but this dataset is top-of-book only and cannot reconstruct deeper price levels.

Depth bands

bookDepth is a sampled aggregate depth surface, not a stream of individual order-book updates:

symbol, event_time, percentage, depth, notional

percentage is Binance's signed distance band around the reference price. Negative bands represent bid-side depth and positive bands represent ask-side depth. In USD-M, depth is base-asset quantity and notional is quote-asset value. In COIN-M, depth is contract quantity and notional is the base-asset equivalent. Join contract specifications when comparing COIN-M depth across symbols or against USD-M.

Liquidation snapshots

COIN-M liquidation rows contain:

symbol, event_time_raw, event_time,
side, order_type, time_in_force,
original_quantity, price, average_price,
order_status, last_filled_quantity, accumulated_filled_quantity

Quantities are contract quantities. side is the side of the forced order: a forced SELL is consistent with a long position being liquidated, while a forced BUY is consistent with a short position being liquidated. These are venue events, not customer identities or a complete account-level liquidation ledger.

Futures metrics

symbol, create_time, source_symbol,
sum_open_interest, sum_open_interest_value,
count_toptrader_long_short_ratio,
sum_toptrader_long_short_ratio,
count_long_short_ratio,
sum_taker_long_short_vol_ratio

These fields are venue aggregates. They do not reveal individual accounts or counterparty identities.

Option EOH surface

symbol, source_date, hour, option_symbol, underlying_symbol,
option_type, expiry_strike,
open, high, low, close, volume_contracts, volume_usdt,
best_bid_price, best_ask_price,
best_bid_quantity, best_ask_quantity,
best_buy_iv, best_sell_iv,
mark_price, mark_iv,
delta, gamma, vega, theta,
open_interest_contracts, open_interest_usdt

EOHSummary ends on 2023-10-23 in Binance's public archive. It is useful for historical option-surface studies but is not a current live options feed. Validate units before interpreting volume_usdt; historical source rows include values that do not behave like dollar premium turnover.

An option identifier such as BTC-230601-26500-P means BTC, expiry 2023-06-01, strike 26,500, put. Preserve the original identifier after parsing it.

Reading from Hugging Face

Install the Python reader dependencies with:

python -m pip install huggingface-hub pyarrow duckdb pytz

Download a compact ticker with the CLI

hf download \
  tmmycruise/autoresearch-crypto-data \
  --repo-type dataset \
  --include 'crypto/binance/v1/market=um/dataset=klines/ticker=BTCUSDT/all_history/**' \
  --local-dir data/binance

If the group is not yet compacted, change the include pattern to:

crypto/binance/v1/market=um/dataset=aggTrades/ticker=BTCUSDT/year=*/period=*/**

Read with PyArrow without downloading the whole repository

import pyarrow.fs as pafs
import pyarrow.parquet as pq
from huggingface_hub import HfFileSystem

hf = HfFileSystem()
arrow_fs = pafs.PyFileSystem(pafs.FSSpecHandler(hf))
revision = "COMMIT_SHA"
path = (
    f"datasets/tmmycruise/autoresearch-crypto-data@{revision}/"
    "crypto/binance/v1/market=um/dataset=klines/"
    "ticker=BTCUSDT/all_history/data.parquet"
)
table = pq.read_table(path, filesystem=arrow_fs)

For reproducible work, put @COMMIT_SHA immediately after the repository ID in an HfFileSystem path, as above, or download with an explicit --revision COMMIT_SHA. Passing revision to the HfFileSystem constructor is not a revision pin.

Query downloaded files with DuckDB

SET TimeZone = 'UTC';

SELECT
    open_time,
    open,
    high,
    low,
    close,
    quote_asset_volume
FROM read_parquet(
    'data/binance/crypto/binance/v1/market=um/dataset=klines/ticker=BTCUSDT/all_history/data.parquet'
)
WHERE open_time >= TIMESTAMPTZ '2024-01-01 00:00:00+00'
  AND open_time <  TIMESTAMPTZ '2024-02-01 00:00:00+00'
ORDER BY open_time;

Set the session timezone before inspecting or exporting TIMESTAMPTZ values. DuckDB stores the instant correctly, but renders it in the session timezone. Explicit UTC avoids a display-time shift on machines configured for another timezone.

For monthly event data, pass a glob and enable schema unification:

SELECT *
FROM read_parquet(
    'data/binance/crypto/binance/v1/market=um/dataset=aggTrades/ticker=BTCUSDT/year=*/period=*/data.parquet',
    union_by_name = true
)
WHERE event_time >= TIMESTAMPTZ '2024-01-01 00:00:00+00'
  AND event_time <  TIMESTAMPTZ '2024-01-02 00:00:00+00';

Point-in-time joins

Point-in-time correctness depends on when a value became observable, not merely the timestamp in its filename.

Availability rules

Use the latest right-hand observation whose safe availability time is less than or equal to the left-hand decision time. Set a maximum tolerated staleness for quotes, metrics, and option snapshots. Do not forward-fill across exchange gaps, listing boundaries, expiry, or reconnect gaps.

DuckDB as-of example

The following joins trade decisions to the last futures metric that was already known:

WITH decisions AS (
    SELECT
        symbol,
        event_time AS decision_time,
        price,
        quantity,
        buyer_is_maker
    FROM read_parquet('aggtrades/*.parquet', union_by_name = true)
), metrics AS (
    SELECT
        symbol,
        create_time AS available_time,
        sum_open_interest,
        sum_open_interest_value,
        sum_taker_long_short_vol_ratio
    FROM read_parquet('metrics/*.parquet', union_by_name = true)
)
SELECT
    d.*,
    m.sum_open_interest,
    m.sum_open_interest_value,
    m.sum_taker_long_short_vol_ratio,
    d.decision_time - m.available_time AS metric_age
FROM decisions d
ASOF LEFT JOIN metrics m
    ON d.symbol = m.symbol
   AND d.decision_time >= m.available_time;

Filter out rows where metric_age exceeds the intended tolerance.

Joining bars without lookahead

A minute bar labeled by open_time = 12:00 contains trades through its close_time. It cannot be used for a decision at 12:00. To use completed-bar features, join decisions against close_time:

SELECT d.*, b.close, b.quote_asset_volume
FROM decisions d
ASOF LEFT JOIN bars b
    ON d.symbol = b.symbol
   AND d.decision_time >= b.close_time;

Cross-market joins

Create an explicit mapping table rather than stripping suffixes inside a research query:

underlying | spot_symbol | usd_m_symbol | coin_m_symbol | option_underlying
BTC        | BTCUSDT     | BTCUSDT      | BTCUSD_PERP   | BTCUSDT
ETH        | ETHUSDT     | ETHUSDT      | ETHUSD_PERP   | ETHUSDT

Version this mapping by validity interval when listings or contract definitions change. A current exchange-information snapshot is reference metadata, not a historical point-in-time listing database.

Joining the historical option surface

Construct an option availability timestamp from source_date and hour. Use a conservative one-hour lag unless the publication time is independently verified. Then as-of join the latest available surface to futures decisions by underlying_symbol. Never use an option row after its expiry, and never carry a surface across a missing-hour gap without flagging its age.

Data-quality rules

• —Use the coverage report and source manifests to identify missing periods.
• —Deduplicate only with the dataset-specific identity key.
• —Keep raw timestamps when diagnosing precision or ordering issues.
• —Treat nullable provider fields as missing, not zero.
• —Reject impossible OHLC rows and required-time nulls.
• —Separate block or liquidation events from ordinary trades where applicable.
• —Do not infer customer or dealer identity from open interest alone.
• —Record the exact Hugging Face revision, input paths, and query in every result.

Source and usage notes

This repository republishes public Binance market-data archives in a research format. Users are responsible for checking Binance's current terms, regional availability, and applicable restrictions. The data are provided for research and may contain source outages, delistings, partial months, or provider corrections.