criptic1/ctrader-ml-systems

1# c-trader ML Systems2 3Three production-grade ML subsystems for the c-trader algorithmic trading platform, integrating the **Aurea architecture** (orthogonal risk-integrated alpha) with passivity-preserving cross-impact theory.4 5## Architecture6 7```8Market Data ──┐9              ▼10┌─────────────────────────┐11│   1. FORECASTING LAYER  │  TimesFM 2.0 + Chronos-2 + Kronos-base12│   Regime-Gated Router   │  with regime detection & fallback chains13└──────────┬──────────────┘14           ▼15┌─────────────────────────┐16│   2. UNCERTAINTY PIPELINE│  ACI conformal prediction + ensemble17│   Calibrated σ̂          │  disagreement + quantile calibration18└──────────┬──────────────┘19           ▼20┌─────────────────────────┐21│   3. CROSS-IMPACT MODEL │  Stieltjes kernel calibration (SDP)22│   Passivity-Preserving  │  + execution optimizer (QP)23└──────────┬──────────────┘24           ▼25┌─────────────────────────┐26│   AUREA ALLOCATOR       │  edge - κ_u·σ̂ - friction27│   Risk Box + Execution  │  + overlap/turnover penalties + cash option28└─────────────────────────┘29```30 31## System 1: Forecasting Layer32 33**Module**: `ctrader_ml.forecasting`34 35Unified multi-model forecaster combining three time-series foundation models with intelligent regime-gated routing:36 37| Model | HF ID | When Used | Key Strength |38|---|---|---|---|39| **Kronos-base** | `NeoQuasar/Kronos-base` (102M) | Trending / High Volatility | Financial-native OHLCV, +93% RankIC vs generic TSFMs |40| **Chronos-2** | `amazon/chronos-2` (120M) | Regime Transitions / Covariates | SOTA zero-shot (90.7% win rate), 8192 context, multivariate |41| **TimesFM 2.0** | `google/timesfm-2.0-500m-pytorch` (500M) | General / Fallback | Fast point forecasts, 2048 context, PyTorch-native |42| **LightGBM/XGBoost** | Your existing models | Mean-reverting / Low Vol | Proven stronger than generic TSFMs on financial returns |43 44### Routing Logic (literature-backed)45- **arxiv:2511.18578** — generic TSFMs underperform tree ensembles on financial returns → keep LightGBM/XGBoost for mean-reverting regimes46- **arxiv:2508.02739** — Kronos: +93% RankIC on financial data → primary model for trending/volatile markets47- **arxiv:2510.15821** — Chronos-2: 90.7% win rate, group attention → best for multi-asset / regime transitions48- **arxiv:2508.02686** — MoE volatility-sensitive routing → gated ensemble architecture49 50### Regime Detector51Uses three orthogonal signals:521. **Realized volatility** (rolling window) — classifies vol regime532. **Hurst exponent** (R/S method) — trending (H>0.6) vs mean-reverting (H<0.4)543. **Directional Movement Index** — trend strength and direction55 56## System 2: Calibrated Uncertainty Pipeline57 58**Module**: `ctrader_ml.uncertainty`59 60Produces σ̂ (the uncertainty penalty) for the Aurea allocator's scoring function:61 62```63net_utility_i = α̂_i − κ_u · σ̂_i − ĉ_i64```65 66### Three-component uncertainty:67 681. **Ensemble Disagreement** (weight=0.30)69   - Normalized std across all model forecasts (TimesFM, Chronos-2, Kronos, LightGBM)70   - Inverse-MSE weighted by recent model performance71 722. **Quantile Width** (weight=0.35)73   - Width of calibrated prediction intervals from Chronos-2 / Kronos MC samples74   - Online isotonic calibration corrects systematic quantile biases75 763. **Conformal Prediction** (weight=0.35)77   - **Adaptive Conformal Inference (ACI)** from Gibbs & Candes (2021)78   - Tracks (1−α̂) quantile of rolling residual window79   - α̂ adapted online: drives empirical coverage to target (e.g., 90%)80   - Handles distribution shifts (regime changes) automatically81 82## System 3: Stieltjes Kernel + Execution Optimizer83 84**Module**: `ctrader_ml.cross_impact` + `ctrader_ml.execution`85 86### Cross-Impact Kernel Calibration87 88Fits the reduced passive cross-impact model:89 90```91G_r(t) = Σ_{k=1}^{K} A_k · exp(−ρ_k · t)92```93 94where each `A_k ≽ 0` (PSD) — **admissibility by construction** (no profitable round-trips).95 96**Calibration pipeline:**971. Empirical kernel estimation from trades/quotes (lagged regression)982. Scalar NNLS warmstart for decay rate initialization993. Matrix SDP fitting with PSD constraints (cvxpy + SCS)1004. Alternating optimization: poles (L-BFGS-B) + residues (SDP)101 102### Passivity Test Suite1035 tests that MUST pass before deployment:104- PSD residue eigenvalues ≥ 0105- Kernel PSD at all time points106- Round-trip cost ≥ 0 (1000 random programs)107- Energy dissipation identity (state-space simulation)108- L¹ approximation error tracking109 110### Execution Optimizer111Multi-asset Almgren-Chriss QP with the passive kernel:112- **Block-QP formulation**: stacked trades V' Q V (DCP-compliant)113- Q is the Toeplitz block-kernel matrix — PSD by Stieltjes theory114- Constraints: participation limits, completion, spread guards115- Feeds friction estimates (ĉ_i) back to the Aurea allocator116 117## Integration: AureaBridge118 119**Module**: `ctrader_ml.integration.aurea_bridge`120 121Orchestrates the full cycle:122 123```python124from ctrader_ml.integration.aurea_bridge import AureaBridge125 126bridge = AureaBridge(127    device="cuda",128    prediction_horizon=24,129    n_assets=5,130    ensemble_mode=True,131    external_models={  # plug in existing c-trader models132        "lightgbm": your_lgbm_predict_fn,133        "xgboost": your_xgb_predict_fn,134    },135)136 137# One-time: calibrate cross-impact kernel138bridge.calibrate_kernel(signed_flow, price_changes, Sigma)139 140# Each trading cycle:141result = bridge.run_cycle(data_dict, Sigma, market_state)142 143# Access outputs:144result.forecasts["AAPL"]          # ForecastResult145result.uncertainties["AAPL"]       # UncertaintyEstimate (σ̂)146result.friction_estimates["AAPL"]  # ĉ147result.sleeve_scores["AAPL"]       # edge - κ_u·σ̂ - ĉ148result.execution_schedules["AAPL"] # optimal trade schedule149 150# Feedback loop (after prices realized):151bridge.update_with_realized("AAPL", y_true=152.5, y_predicted=153.0)152```153 154## Installation155 156```bash157# Core dependencies (always needed)158pip install numpy pandas scipy cvxpy[scs,osqp]159 160# For TimesFM 2.0161pip install timesfm[torch]162 163# For Chronos-2164pip install "chronos-forecasting>=2.0"165 166# For Kronos167git clone https://github.com/shiyu-coder/Kronos && cd Kronos && pip install -r requirements.txt168```169 170## Tests171 172```bash173cd /path/to/project174python ctrader_ml/tests/test_all_systems.py175# Expected: 31 passed, 0 failed, 0 skipped176```177 178## File Structure179 180```181ctrader_ml/182├── forecasting/183│   ├── regime_detector.py       # Hurst + ADX + vol regime classification184│   ├── model_registry.py        # Lazy-loading model management185│   └── unified_forecaster.py    # Regime-gated multi-model router186├── uncertainty/187│   └── calibrated_uncertainty.py  # ACI + ensemble + quantile calibration188├── cross_impact/189│   └── stieltjes_kernel.py      # SDP calibration + passivity tests190├── execution/191│   └── cross_impact_executor.py # Block-QP optimizer + friction estimator192├── integration/193│   └── aurea_bridge.py          # Full-cycle orchestrator + allocator scorer194├── utils/195│   └── types.py                 # Shared data types and enums196└── tests/197    └── test_all_systems.py      # 31-test comprehensive suite198```199 200## Key References201 202| Paper | What It Provides |203|---|---|204| [arxiv:2511.18578](https://arxiv.org/abs/2511.18578) | Generic TSFMs underperform tree ensembles on financial returns |205| [arxiv:2508.02739](https://arxiv.org/abs/2508.02739) | Kronos: financial-native OHLCV foundation model |206| [arxiv:2510.15821](https://arxiv.org/abs/2510.15821) | Chronos-2: SOTA zero-shot with group attention |207| [arxiv:2310.10688](https://arxiv.org/abs/2310.10688) | TimesFM: decoder-only time-series foundation model |208| [arxiv:2508.02686](https://arxiv.org/abs/2508.02686) | MoE volatility-sensitive routing |209| Gibbs & Candes (2021) | Adaptive Conformal Inference (ACI) |210| Almgren & Chriss (2000) | Optimal execution with market impact |211| Uploaded study | Passivity-preserving cross-impact reduction |212| Aurea transcript | Orthogonal Risk Integrated Alpha architecture |213