XinranSong/nanojev-coinbase-btc-direction-v2

Coinbase BTC historical midpoint direction — v2

15,089 real market decision times, with three direction questions per time (45,267 correlated questions): 1, 5 and 15 minutes. No synthetic market rows or generated prices are used.

Source and license

Derived from Martin Søgaard Nielsen (martinsn), High Frequency Crypto Limit Order Book Data, version 1, BTC_1min.csv. The publisher reports Coinbase websocket collection and declares CC0: Public Domain. The source contains 17,113 rows from 2021-04-07 11:33:41 UTC to 2021-04-19 09:54:00 UTC (13 UTC dates, about 12 elapsed days). No ETH or additional dates are included.

The unchanged source ZIP and provenance are included under trading/. Original source and derived dataset records are distributed under CC0-1.0, including any rights in our dataset-specific transformations. See LICENSE and CC0 legal text. Newly authored software uses MIT (LICENSE-CODE); vendored NanoJev software retains its own MIT notice. The source's license declaration was rechecked through the public Kaggle API on 2026-09-20. This states the publisher's declaration, not an independent certification by Coinbase. Source attribution and collection-tool acknowledgment are preserved in THIRD_PARTY_NOTICES.md.

Inputs and observed targets

State summarizes the latest completed midpoint, spread, top-five bid/ask book-notional imbalance and historical returns over a 15-row lookback. It records actual elapsed seconds for historical offsets. It excludes flow columns, future prices, centered windows, interpolation and full-dataset fitted normalization.

For source timestamp si, assumed availability is ai = si + 60 seconds. Decision time t = ai. For each h in {60,300,900}, choose the first row with aj >= t+h, allowing at most 15 additional seconds. Compute r = 10000*(mj/m_i-1): down if r < -2 bps, up if r > +2 bps, otherwise flat (both boundaries included).

gold holds realized direction labels. gold_label_kind is observed_outcome; gold_probs and gold_probs_kind are empty mappings because a realized future outcome is not a known conditional distribution. No hindsight buy/sell or execution-profit labels are created.

The source does not fully specify minute aggregation endpoints. The 60-second delay is a conservative adapter assumption, not a proof of original timestamp semantics. Further source collection-code auditing is required for stronger real-time claims.

Chronological splits and leakage controls

All dates below are UTC in April 2021; intervals are left-closed/right-open.

The full support window from the earliest historical source timestamp to the latest of all three label availability times must fit inside a single split, with a 60-second boundary embargo (strict upper bound). Any gap exceeding 90 seconds across history through the final target removes the whole state. Invalid targets remove all three questions together. Source groups are UTC calendar days, with every horizon from a state kept together.

2,024 rows are excluded: 14 insufficient history, 678 target availability/tolerance failures, 1,244 excessive gaps, 88 boundary/embargo failures. Adjacent windows within each split remain highly dependent. ood is a final temporal holdout; it does not establish a distinct market regime. All labels can be replayed from original source line numbers and metadata.

Intended use and limitations

For historical direction-classification experiments, with accuracy, log loss and Brier score reported separately per horizon. A single asset and short 2021 period cannot establish market generalization or tradable profitability. No fees, slippage or execution model is supplied. The three horizons are correlated, not three times as much independent market data. v1 and v2 overlap in source windows and should not be merged as independent samples.

Format and loading

The authoritative NanoJev records are trading/data/{split}.jsonl. Each line is one state with multiple questions. Original bytes, mappings and empty dictionaries are preserved. Native fields are id, state_id, family_id, split, state, questions, gold, gold_probs, gold_probs_kind, gold_label_kind, and metadata.

For the Hugging Face viewer and datasets, viewer/*.parquet provides one row per native record. Mapping fields are losslessly serialized as questions_json, gold_json, gold_probs_json, gold_probs_kind_json, gold_label_kind_json, and metadata_json. This avoids Arrow empty-struct/schema issues and preserves dictionary keys exactly. These are an alternate representation of the SAME records, not additional training data. source_group_id is also exposed as a convenience column.

import json
from datasets import load_dataset

repo_id = "XinranSong/nanojev-coinbase-btc-direction-v2"
ds = load_dataset(repo_id, split="train")
preview_row = ds[0]
row = {k: preview_row[k] for k in ["id", "state_id", "family_id", "split", "state"]}
for key in ["questions", "gold", "gold_probs", "gold_probs_kind", "gold_label_kind", "metadata"]:
    row[key] = json.loads(preview_row[key + "_json"])

For NanoJev, download the native records directly; no conversion is needed:

from huggingface_hub import snapshot_download
folder = snapshot_download(
    repo_id="XinranSong/nanojev-coinbase-btc-direction-v2", repo_type="dataset",
    allow_patterns=["trading/data/*"],
)
print(folder + "/trading/data")  # pass this directory to the trainer --input

Using a complete NanoJev checkout and its training dependencies:

python scripts/train_pipeline_decisions.py --input /path/to/trading/data --validate-only
python scripts/train_pipeline_decisions.py --input /path/to/trading/data \
  --objective observed_outcome --loss ce --max-length 512 --output-dir runs/trading_v2

Model input is only `state` and `questions`. Never append `gold`, `gold_probs`, or `metadata` to the prompt. In particular, Trading metadata contains future prices for label auditing. Use only train for parameter training, dev for model selection, calibration for fitting calibration parameters, and keep test/ood for final evaluation. Do not concatenate these splits or treat the viewer as extra data.

Reproduction and validation

Python 3.10+ standard library is sufficient for the generator/adapter. Run from the repository root:

python trading/build_trading.py
python vendor/nanojev/train_pipeline_decisions.py --input trading/data --validate-only
python verify_release.py

verify_release.py needs pyarrow for lossless preview round-trip checks. requirements-preview.txt lists the versions tested for publication. Generation does not require the preview dependencies. build_preview.py rebuilds the Parquet representation after regenerating native data.

The original v2 package passed 14 regression tests, including numerical checks and Trading future-perturbation/boundary checks. Historical tokenization verification at NanoJev commit 71a513bb0163b5634467842b523ee0c0ed6fb1c7 loaded all questions with the official Qwen3-0.6B tokenizer at max length 512. See verification/token_audit_original.json; this is a historical loader test, not model training. The vendored validator remains pinned to that original commit, with its MIT license.

All 15,089 native records and 45,267 question targets also passed the current upstream reader/validator/objective checks at commit 618cea6d906d54e128360786d12f703fff2b1245 (2026-09-20). This is interface compatibility, not a training-performance result. Publication-specific validation and the upstream contract check are recorded in verification/publication_checks.json. No trained weights or controlled training benchmark are included. An informal mixed-task training attempt was reported to reduce performance on other gaming tasks; run settings, metrics and independent reproduction are unavailable. This is anecdotal feedback, not evidence identifying the cause or establishing that the dataset is universally harmful or beneficial. Evaluate each task independently and tune mixture weights on dev if combining datasets.

This is a community dataset prepared for use with NanoJev, not an official NanoJev release. File hashes are in SHA256SUMS.json (excluding that manifest itself).