thesven/fintime-decoder-dataset

Dataset Card for FinTime Dataset

Dataset Summary

FinTime Dataset is a comprehensive, large-scale financial time series dataset designed for training and evaluating decoder-only models on financial forecasting tasks across diverse asset classes and market conditions. Composed of real-world market data spanning equities, cryptocurrencies, forex, commodities, and indices, the dataset captures the complexity, volatility, and multi-scale dynamics typical of financial markets. Enhanced with 80+ technical indicators and statistical features, FinTime provides a rich foundation for developing and benchmarking time series foundation models in financial domains.

The dataset covers 697 carefully curated financial instruments from January 2010 to July 2025, encompassing major market cycles including the 2020 pandemic crash, crypto boom/bust cycles, and various economic regimes. Unlike synthetic or limited-scope financial datasets, FinTime reflects the full spectrum of market behaviors observed across different asset classes and time periods.

FinTime consists of approximately 2-5 million sequences across 697 tickers, with each sequence containing 64 timesteps and 87 features. The dataset employs a sliding window approach with configurable overlap to maximize training data while maintaining temporal consistency. Each sequence represents a comprehensive view of market dynamics through multiple lenses:

• —Price Action: OHLCV data capturing basic market movements
• —Momentum Indicators: RSI, Stochastic, Williams %R revealing market sentiment
• —Trend Analysis: Moving averages, MACD, ADX identifying directional bias
• —Volatility Measures: ATR, Bollinger Bands, VIX-like indicators capturing risk
• —Volume Dynamics: VWAP, OBV, accumulation/distribution patterns
• —Statistical Features: Z-scores, percentile ranks, higher-order moments
• —Temporal Context: Calendar effects, seasonality, time-based features

Key characteristics that make FinTime particularly challenging and realistic:

• —Regime Changes: Captures bull/bear markets, crisis periods, and structural shifts
• —Cross-Asset Correlations: Includes correlated and uncorrelated instrument pairs
• —High-Frequency Patterns: Intraday volatility, gap dynamics, and microstructure effects
• —Non-Stationarity: Evolving market relationships and changing volatility regimes
• —Heavy-Tailed Distributions: Extreme events, fat tails, and asymmetric returns
• —Multi-Scale Dependencies: Short-term noise, medium-term trends, long-term cycles

Evaluating Models on FinTime

We provide comprehensive evaluation frameworks and baseline implementations; see the code repository.

Dataset Structure

Each entry in the dataset consists of:

• —A multivariate time series with 87 features across 64 timesteps (daily frequency)
• —Comprehensive metadata including timestamps, sampling information, and asset classification
• —Target variables for next-period forecasting (price, returns, direction)
• —Feature attribution for interpretability and ablation studies

Data Schema (BOOM-Compatible Format)

{
  "series": [[...], [...], ...],           // 87 features × 64 timesteps
  "timestamps": [1577836800, ...],         // Unix timestamps  
  "sequence_length": 64,
  "num_variates": 87,
  
  "series_id": "AAPL_0",
  "sampling_start_time": 1577836800.0,
  "sampling_frequency": "daily",
  "domain": "financial",
  "metric_type": "equity",
  "subcategory": "sp500",
  
  "ticker": "AAPL",
  "window_start_date": "2023-01-01", 
  "window_end_date": "2023-03-05",
  
  "target_next_close": 150.25,
  "target_next_return": 0.0123,
  "target_direction": 1,
  
  "feature_names": ["open", "high", "low", ...]
}

Feature Categories

Collection and Sources

Data is sourced from Yahoo Finance API, providing comprehensive coverage of global financial markets. The collection process employs:

• —Rate-Limited Retrieval: Respectful API usage with configurable delays
• —Quality Validation: Automated checks for data completeness and consistency
• —Error Handling: Robust retry logic and missing data interpolation
• —Incremental Updates: Efficient data refresh and historical backfill capabilities

The preprocessing pipeline includes:

• —Data cleaning and outlier detection (>50% daily moves filtered)
• —Technical indicator calculation using pandas-ta library
• —Feature normalization via StandardScaler/MinMaxScaler
• —Sequence generation with overlapping windows
• —Target variable engineering for forecasting tasks

Asset Universe Coverage

Comparison with Other Financial Datasets

FinTime differs significantly from traditional financial benchmarks in several key dimensions:

• —Scale: 2-5M sequences vs. typical datasets with <100K samples
• —Feature Richness: 87 engineered features vs. basic OHLCV (5 features)
• —Asset Diversity: 697 instruments across 6 asset classes vs. single-asset focus
• —Temporal Coverage: 15+ years including multiple market regimes vs. limited periods
• —Format Compatibility: BOOM-style format optimized for foundation models

Statistical analysis reveals FinTime's unique characteristics:

• —Higher spectral entropy indicating complex temporal dynamics
• —Non-Gaussian return distributions with heavy tails and skewness
• —Regime-dependent correlations and time-varying volatility
• —Multi-scale dependencies from intraday to multi-year cycles

Model Training Considerations

Decoder-Only Architecture Benefits

• —Autoregressive Generation: Natural fit for sequential financial data
• —Attention Mechanisms: Capture long-range dependencies and regime changes
• —Scalability: Efficient training on large sequence datasets
• —Transfer Learning: Pre-trained models can adapt to new assets/markets

Evaluation Metrics

• —Next-Step Prediction: Accuracy for price/return forecasting
• —Directional Accuracy: Binary classification performance
• —Risk-Adjusted Returns: Sharpe ratio, maximum drawdown
• —Regime Robustness: Performance across different market conditions

Baseline Models

We provide implementations and benchmarks for:

• —Statistical Models: ARIMA, GARCH, Vector Autoregression
• —Machine Learning: Random Forest, XGBoost, LightGBM
• —Deep Learning: LSTM, Transformer, specialized time series architectures
• —Foundation Models: GPT-style decoders, time series transformers

Usage Examples

Loading the Dataset

from datasets import load_dataset

# Load complete dataset
dataset = load_dataset("your-username/fintime-dataset")

# Access splits
train_data = dataset["train"] 
val_data = dataset["validation"]
test_data = dataset["test"]

# Filter by asset class
equity_data = dataset.filter(lambda x: x["metric_type"] == "equity")
crypto_data = dataset.filter(lambda x: x["metric_type"] == "crypto")

PyTorch Integration

import torch
from torch.utils.data import DataLoader

class FinTimeDataset(torch.utils.data.Dataset):
    def __init__(self, hf_dataset, sequence_length=64):
        self.hf_dataset = hf_dataset
        self.sequence_length = sequence_length
    
    def __getitem__(self, idx):
        sample = self.hf_dataset[idx]
        
        # Extract features and targets
        features = torch.tensor(sample['series'], dtype=torch.float32)  # [87, 64]
        target_return = torch.tensor(sample['target_next_return'], dtype=torch.float32)
        target_direction = torch.tensor(sample['target_direction'], dtype=torch.long)
        
        return {
            'features': features.transpose(0, 1),  # [64, 87] for transformer input
            'target_return': target_return,
            'target_direction': target_direction,
            'timestamps': torch.tensor(sample['timestamps']),
            'ticker': sample['ticker']
        }

# Create DataLoader
dataset = FinTimeDataset(train_data)
dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

Streaming for Large-Scale Training

# Memory-efficient streaming
dataset = load_dataset("your-username/fintime-dataset", streaming=True)

# Process in chunks
for batch in dataset["train"].iter(batch_size=1000):
    # Mini-batch training
    features = torch.stack([torch.tensor(item['series']) for item in batch])
    targets = torch.tensor([item['target_next_return'] for item in batch])
    
    # Training step
    optimizer.zero_grad()
    predictions = model(features)
    loss = criterion(predictions, targets)
    loss.backward()
    optimizer.step()

Links

• —Code Repository
• —Model Benchmarks
• —Technical Documentation
• —Dataset Generator Pipeline

Citation

@dataset{fintime2025,
  title={FinTime Dataset: Large-Scale Financial Time Series for Foundation Model Training},
  author={Claude Code},
  year={2025},
  publisher={Hugging Face},
  url={https://huggingface.co/datasets/your-username/fintime-dataset},
  note={Generated with Claude Code}
}

License

This dataset is released under the Apache License 2.0. The underlying financial data is sourced from Yahoo Finance, which provides this data for educational and research purposes.

Acknowledgments

• —Yahoo Finance for providing comprehensive market data access
• —pandas-ta library for robust technical indicator implementations
• —Hugging Face for dataset infrastructure and hosting
• —BOOM dataset authors for establishing evaluation frameworks for time series foundation models
• —The quantitative finance community for domain expertise and validation

Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>