Traders-Lab/TroveLedger

1---2language:3- en4pretty_name: TroveLedger Financial Time Series Dataset5task_categories:6- time-series-forecasting7- tabular-regression8tags:9- finance10- financial11- yahoo-finance12- stock-market13- stocks14- OHLC15- time-series16- trading17- equities18- indices19- historical-data20- stock-data21- market-data22- finance-data23- yfinance24dataset_info:25  features:26  - name: symbol27    dtype: string28  - name: time29    dtype: int6430  - name: open31    dtype: float6432  - name: high33    dtype: float6434  - name: low35    dtype: float6436  - name: close37    dtype: float6438  - name: volume39    dtype: int6440  splits:41  - name: daily42    num_examples: null43  - name: hourly44    num_examples: null45  - name: minute46    num_examples: null47size_categories:48- n<1K49- 1K<n<10K50- 10K<n<100K51- 100K<n<1M52- 1M<n<10M53- 10M<n<100M54license: cc-by-nc-sa-4.055---56 57> 🔄 **Data Freshness & Official Source Notice**  58> This dataset is actively maintained and updated (often daily) at its **only official source**:  59> 👉 **[Traders-Lab/TroveLedger](https://huggingface.co/datasets/Traders-Lab/TroveLedger)**  60>  61> *If you are reading this on a different repository (e.g., a fork, mirror, or third-party copy), please be aware that you are looking at a static, outdated snapshot. Financial time-series data loses its value quickly without regular updates. For the latest, most accurate data, always refer back to the official [Traders-Lab/TroveLedger](https://huggingface.co/datasets/Traders-Lab/TroveLedger) repository.*62 63> ⚠️ **Data Source & Liability Disclaimer**  64> This dataset contains historical financial time series data aggregated via the `yfinance` Python library (sourced from Yahoo Finance).  65> - **No Ownership:** The underlying raw financial data remains the property of its respective original sources.  66> - **Usage:** This dataset is provided strictly for **non-commercial research and educational purposes**.  67> - **No Warranty:** Provided "AS IS", without warranty of any kind. USE AT YOUR OWN RISK.  68> - **Commercial Use:** If you intend to use this data commercially, it is your sole responsibility to consult the Terms of Service of Yahoo Finance and other data providers directly.  69 70# 🗃️ TroveLedger — Financial Time Series Dataset71 72![TroveLedger Banner](media/Banner-TroveLedger.jpg)73 74**A growing ledger of accumulated market history.**75 76> ### ⚠️ Temporary Notice: Intraday Data Adjustments (January 2026)77>78>**What happened:**  79> A discrepancy has been identified in the minute- and hourly-resolution data: these series are currently **not fully adjusted** for stock splits and dividends. Daily-resolution data remains correctly adjusted (as provided by the source).80>81> **Why this matters:**  82> For accurate backtesting and model training – especially in intraday strategies – proper adjustments are essential to avoid distorted price histories.83>84> **What I'm doing:**  85> I am actively working on a comprehensive fix to bring all resolutions to full adjustment consistency. This will ensure the highest possible data quality moving forward.86>87> **Short-term plan:**  88> Until the fix is complete, **no new indices or symbols will be added** to avoid introducing further unadjusted data.  89>90> If the full correction takes longer than expected, an intermediate step may be applied: removing affected (unadjusted) intraday segments to keep the published dataset accurate, even if temporarily reduced in scope. (***update January 13.: Fix is in place, there seems no need for this***)91>92> ### **Current Status (updated January 19, 2026):**  93> The adjustment logic for splits and dividends in minute- and hourly data is showing promising results in testing. > It detects and applies known splits correctly and handles most dividend events sensibly.94> However, the details are taking more time than expected.95>96> **Challenges & next steps:**97> - Some noise/false positives appear in dividend detection (adjustments applied where none occurred).98> - Fine-tuning thresholds and filters is ongoing to reduce this noise while preserving real events.99> - This requires additional experimentation and manual verification over the weekend.100>101>**Progress summary:**102> - Core fix works in principle → clear improvement visible on tested samples103> - Full-dataset re-processing continues in background104> - Still in validation phase – no re-publish until confident in the quality105>106>**Important reminder:**  107> Until the fix is validated and re-published, **do not use the current intraday data** for strategies/backtests that depend on accurate split/dividend adjustments.108>109> **Ongoing operations:**  110> Daily OHLCV accumulation for existing symbols continues normally (no gaps).  111> No new indices/symbols are being added during this phase.112>113> Updates will be posted here and on X (@TroveLedger) as soon as next milestones are reached (threshold tuning done, validation samples passed, reduced/clean version live, full version live).114>115> Thank you for your continued patience — this extra refinement will make the dataset noticeably more reliable for serious AI trading bot training and quant work.116 117 118![Error](media/Error.jpg)119 120---121### 🔔 **Latest Dataset Update**122 123**Date:** 2026-01-06  124**New addition:**  🇧🇪 BEL 20 (Belgium)  125> Today, the dataset expands with the BEL 20 — Belgium’s primary equity index and the central barometer of its domestic capital market.  126> Composed of the 20 most liquid and heavily traded companies listed in Brussels, the BEL 20 reflects an economy shaped by finance, consumer goods, industry, and cross-border European integration.127>128> 📊 **Market context:**  129> **Region:** Europe — Belgium  130> **Scope:** Large-cap benchmark (BEL 20 constituents)  131> **Sector exposure:** Financials, Consumer Staples, Industrials, Utilities  132> **Data coverage:** Minute, hourly, and daily OHLC data133>134> 📈 **What this means:**  135> A compact, liquid market segment that adds depth to Western European coverage with consistent, high-resolution time series.136>137> 🔜 **What’s next:**  138> Continued expansion across international markets, preserving uniform structure and historical depth.139 140<details>141<summary>142Click to expand143</summary>144 145* [Download symbol list (BEL20.txt)](Symbols/BEL20.txt)146 147<img src="media/BEL20.jpg"/>148 149* More information about previous additions can be found further below in <a href="#growing-treasury">The Growing Treasury</a>150 151</details>152 153> ### Recent Index Additions154>155> | Date       | Index                 | Region       | Symbols |156> |------------|-----------------------|--------------|---------|157> | 2026-01-06 | BEL 20                | Belgium 🇧🇪  | 20      |158> | 2026-01-05 | DAX | Germany 🇩🇪 | 40 |159> | 2026-01-02 | ASX 200      | Australia 🇦🇺 | 200 |160> | 2025-12-30 | OMX Stockholm 30       | Sweden 🇸🇪 | 30 |161> | 2025-12-29 | TSX (S&P/TSX Composite) | Canada 🇨🇦 | 222 |162> | 2025-12-24 | SMI             | 🇨🇭 Switzerland | 20     |163> | 2025-12-23 | NIFTY 50        | 🇮🇳 India | 50     |164> | 2025-12-22 | FTSE 100        | 🇬🇧 United Kingdom | 100     |165> | 2025-12-19 | S&P 500         | 🇺🇸 US | 503     |166> | 2025-12-18 | Hang Seng Index | 🇭🇰 Asia | 82      |167> | 2025-12-17 | EURO STOXX 50   | 🇪🇺 Europe | 50      |168 169---170 171## 📌 Overview172 173**TroveLedger** is a public financial time series dataset focused on **long-term accumulation of high-quality intraday data**.174 175The dataset provides OHLC and volume data at multiple time resolutions and is designed primarily for **machine learning, quantitative research, and systematic trading experiments**.176 177Unlike many freely available data sources, TroveLedger emphasizes **continuity over time**, especially for minute-level data.178 179### Scale & Granularity180- Total: Over 40 million rows across all symbols and resolutions (growing rapidly)181- Per symbol: Varies significantly – from <1,000 rows (young stocks, daily) to >500,000 rows (established stocks, minute-resolution)182- Ideal for both focused single-symbol training and large-scale multi-market models183 184## 🔑 What makes TroveLedger different185 186High-resolution intraday data is difficult to obtain from free sources over extended periods.187 188Typical public data access (e.g. via yfinance) provides:189 190* **Daily candles:** often spanning decades191* **Hourly candles:** roughly one year into the past192* **Minute candles:** usually limited to the most recent 7 days193 194Repeatedly downloading rolling 7-day windows results in **short, fragmented histories** that are poorly suited for training models on intraday behavior.195 196TroveLedger takes a different approach:197 198* Minute-level data is **accumulated continuously**199* Time series are **extended, not replaced**200* Over time, this results in **months of gap-free minute data per instrument**201 202This accumulated depth forms a substantially more reliable foundation for intraday research and model training.203 204> 🧱 **Data Integrity Philosophy**205>206> TroveLedger prioritizes *continuity over frequency*.  207> The primary goal is not to fetch data as often as possible, but to ensure that once a time series starts, it remains **gap-free**.208>209> Minute-level data is accumulated incrementally over time, creating long, uninterrupted histories that are not obtainable from fresh API queries alone.210>211> This makes the dataset particularly suitable for model training, backtesting, and regime analysis.212 213 214## 📦 Dataset Structure215 216The dataset is organized as follows:217 218- **/data/{category}/{symbol}/{symbol}.{interval}.valid.parquet**219 220Where:221- `{category}`: e.g., `equities/us`, `indices/sp500`, `indices/eurostoxx50` (growing with new indices)222- `{symbol}`: Stock ticker (e.g., AAPL, BMW.DE)223- `{interval}`: One of `days` (daily), `hours` (hourly), or `minutes` (1-minute)224 225The `.valid` suffix indicates that these files have passed quality checks and are ready for use. Only these cleaned, validated files are included in the dataset – temporary or intermediate files from the pipeline are excluded.226 227**Tip for users**: The `.valid` part is intentionally kept as a flexible "state" marker. You can easily rename or copy files to add your own states (e.g., `.train.parquet` or `.test.parquet`) for train/validation/test splits in your ML workflows. This pattern makes it simple to organize experiments without changing the core data.228 229### Data Instances230 231Here's an example row from a typical daily Parquet file (e.g., for AAPL.days.valid.parquet):232 233| symbol | time       | open   | high   | low    | close  | volume    |234|--------|------------|--------|--------|--------|--------|-----------|235| AAPL  | 1704067200 | 192.28 | 192.69 | 191.73 | 192.53 | 42672100 |236 237- `time` is a Unix timestamp (e.g., 1704067200 = January 1, 2024, 00:00 UTC).238- All prices are in the symbol's native currency (e.g., USD for US equities).239 240### Dataset Creation241 242### Curation Rationale243TroveLedger was created to provide a reliable, expanding source of historical OHLCV data for AI-driven trading research, addressing gaps in continuity and international coverage.244 245### Source Data246All data is sourced from Yahoo Finance via the `yfinance` Python library. Index components are automatically extracted from Wikipedia pages using a custom API-based pipeline for sustainability.247 248### Data Collection and Processing249- Symbols are selected from major indices (e.g., S&P 500, EURO STOXX 50) and equities.250- Data is fetched at daily, hourly, and 1-minute resolutions, validated for completeness, and stored in Parquet format for efficiency.251- Quality checks remove gaps or anomalies; only ".valid" files are included.252- Updates occur periodically to extend histories and add new indices based on community input.253 254### Who are the source data producers?255Yahoo Finance (public market data). No personal data is included.256 257## 🔄 Update Philosophy258 259The primary objective is **data continuity**, not guaranteed daily updates.260 261In particular:262 263* Daily updates are **not guaranteed**264* Preventing **gaps in accumulated minute data** has priority265* Updates are performed on trading days whenever possible266 267Minute data is updated most frequently to ensure continuity.268 269Hourly and daily data are updated on a **rotation basis** to reduce unnecessary repeated downloads and to remain considerate of public data sources.270These datasets are guaranteed to be **no older than one week**.271 272For most training scenarios, this is fully sufficient.273When models are deployed in real-world environments, current market data is typically provided directly by the target trading platform.274 275## 📈 Scope & Growth276 277TroveLedger started with a curated universe of approximately 500 equities inherited from earlier *Preliminary* datasets.278 279Going forward:280 281* Entire indices are added step by step282* The covered universe will grow continuously283* Expansion is performed incrementally to ensure data integrity and operational stability284 285This gradual approach allows issues to be detected early and handled without disrupting existing data.286 287## 🎯 Intended Uses288 289- **Primary Use**: Training and evaluating machine learning models for trading strategies and autonomous AI bots.290- **Other Uses**: Time series analysis, financial research, educational projects, and community-driven extensions.291 292TroveLedger is suitable for:293 294* machine learning on financial time series295* intraday and swing trading research296* feature engineering on OHLC data297* backtesting strategies requiring dense intraday history298* exploratory quantitative analysis299 300## ⚠️ Limitations & Notes on Data Sources301 302- **Data Freshness**: Data is typically a few days old, not real-time.303- **Coverage**: Not all symbols may have complete historical data, especially for minute-resolution or newly added indices.304- **Growth Phase**: The dataset is actively expanding; check for updates on new indices and symbols.305- **Not financial advice**: This dataset is for research and educational purposes only. Past performance is no guarantee of future results.306 307Data is derived from publicly accessible market data sources (e.g. via yfinance).308 309While care is taken to ensure consistency and continuity, this dataset is provided **as-is** and without guarantees regarding completeness or correctness.310 311Users are responsible for verifying suitability for their specific use cases and for complying with the terms of the original data providers.312 313## 📜 License & Usage314 315This dataset is provided **solely for non-commercial research and educational purposes**.316 317The data is retrieved from public sources via the yfinance library (Yahoo Finance). All rights remain with the original data providers.318 319Redistribution of this dataset is **not permitted** without explicit permission from the original sources.320 321See the [`LICENSE`](./LICENSE) file for full details.322 323NO WARRANTY IS PROVIDED. Use at your own risk.324 325## 💬 Feedback, Suggestions & Community Support326 327TroveLedger is a growing, community-driven project providing high-quality OHLCV data for training AI models on financial markets and trading strategies. Your input makes it better!328 329- **What are you building?** I'd love to hear how you're using TroveLedger! Share your projects, trading bot ideas, ML models, or research directions – it motivates me to keep expanding and might inspire others.330- **Desired indices**: Which major indices are you waiting for most? I'll prioritize based on demand and feasibility.331- **Helping expand indices**: The pipeline uses the Wikipedia API to automatically extract components. It works best with a structured table containing both company names and clean, yfinance-compatible ticker symbols.332  - Simply share the Wikipedia page URL (any language) for your desired index.333  - If the table needs tweaks (e.g., missing or unclear ticker column, prefixes in symbols), improving it on Wikipedia is the most sustainable way – the global community then keeps it updated long-term!334  - Once ready, post the link here, and I'll integrate it quickly.335 336Interested in a deeper dive into the exact table format and config options my pipeline supports (with examples like zero-padding, suffixes, or language overrides)? Let me know – if there's demand, I'll create a dedicated guide soon!337 338Join the discussion in [Hugging Face Discussions](https://huggingface.co/datasets/Traders-Lab/TroveLedger/discussions).339 340---341 342### 🏛️ The Growing Treasury343<div id="growing-treasury">344 345    Watch TroveLedger expand across global markets – a visual chronicle of added indices:  346 347<details>348<summary>🇧🇪 BEL 20 (January 6, 2026) – Measured Wealth in a Small Vault</summary>349 350Behind heavy stone walls and careful accounting, the **BEL 20** represents a market where scale is limited but liquidity is deliberate.351Belgium’s leading financial, industrial, and consumer firms form a tightly curated index, reflecting a capital market defined by restraint, stability, and European connectivity.352 353This addition strengthens TroveLedger’s continental European archive, capturing a compact exchange whose value lies not in breadth, but in precision and consistency across timeframes.354 355* [Download symbol list (BEL20.txt)](Symbols/BEL20.txt)356 357<img src="media/BEL20.jpg"/>358</details>359 360<details>361<summary>🇩🇪 DAX (January 5, 2026) – Engineered Capital at Industrial Scale</summary>362 363At the intersection of precision engineering and global trade, the **DAX** captures Germany’s most influential publicly listed corporations.364Industrial manufacturers, automotive leaders, chemical groups, and financial institutions dominate the index, forming a market shaped by export depth and operational discipline.365 366This entry anchors TroveLedger’s core European coverage, adding a market where long-cycle industry, efficiency, and global exposure define capital behavior across timeframes.367 368- [Download symbol list (DAX.txt)](Symbols/DAX.txt)369 370<img src="media/DAX.jpg"/>371</details>372 373<details>374<summary>🇦🇺 ASX 200 (January 2, 2026) – Weighed by Earth and Capital</summary>375 376Across vast distances and resource-rich ground, the **ASX 200** captures an equity market anchored in tangible assets and institutional capital.377Mining conglomerates, major banks, energy firms, and healthcare leaders dominate the index, forming a market profile distinct from technology-heavy regions.378 379This addition extends TroveLedger’s reach into Oceania, preserving a market where commodities, yield, and global demand cycles leave clear historical traces across intraday and long-term data.380 381- [Download symbol list (ASX200.txt)](Symbols/ASX200.txt)382 383<img src="media/ASX200.jpg"/>384</details>385 386 387<details>388<summary>🇸🇪 OMX (December 30, 2025) – Order in the Nordic Ledger</summary>389 390In the measured calm of Northern Europe, the **Stockholm Stock Exchange (OMX)** records value through discipline, transparency, and long-term orientation.391Industrial groups, financial institutions, and globally oriented consumer firms dominate the OMX Stockholm 30, forming a compact yet internationally relevant market profile.392 393This entry adds a distinctly Nordic balance to TroveLedger — one shaped by export strength, institutional stability, and methodical capital allocation across intraday and long-horizon views.394 395- [Download symbol list (OMX.txt)](Symbols/OMX.txt)396 397<img src="media/OMX.jpg"/>398</details>399 400<details> <summary>🇨🇦 TSX (December 29, 2025) – Beneath the Surface of Canadian Capital </summary>401 402Deep underground, where resources are extracted and value is carefully recorded, the **Toronto Stock Exchange (TSX)** reflects the structural foundations of the Canadian economy.403Banks, miners, energy producers, and industrial firms form the backbone of the S&P/TSX Composite, making it a distinctive counterweight to more tech-heavy global indices.404 405This entry extends TroveLedger’s North American coverage beyond the United States, adding a market shaped by commodities, capital discipline, and long-cycle industries — all captured across consistent intraday and long-horizon timeframes.406 407- [Download symbol list (TSX.txt)](Symbols/TSX.txt)408 409<img src="media/TSX.jpg"/>410</details>411 412<details>413<summary>🇨🇭 SMI (December 24, 2025) – Alpine quality meets market stability</summary>414 415The **Swiss Market Index (SMI)** has been added to TroveLedger, bringing the premier blue-chip index of Switzerland into our global dataset.  416Representing 20 of the largest and most liquid companies listed on the SIX Swiss Exchange — including giants like Nestlé, Roche, and Novartis — the SMI offers a unique exposure to one of the world’s most stable and innovation-driven economies.417 418The SMI reflects Switzerland’s enduring role as a benchmark for quality, resilience, and long-term value.419 420- [Download symbol list (SMI.txt)](Symbols/SMI.txt)421 422<img src="media/SMI.jpg" alt="TroveLedger as Santa Claus riding a golden sleigh filled with gold coins and gifts through snowy Swiss Alps, with a Swiss flag flying, next to a treasure chest labeled 'SMI'" style="max-width:100%;"/>423</details>424 425<details>426<summary>🇮🇳 NIFTY 50 (December 23, 2025) – India takes center stage</summary>427 428The **NIFTY 50 Index** from India has been incorporated into TroveLedger, enriching the dataset with one of South Asia’s most referenced equity benchmarks. It represents 50 of the largest and most liquid Indian stocks listed on the National Stock Exchange.429 430- [Download symbol list (NIFTY.txt)](Symbols/NIFTY.txt)431 432<img src="media/NIFTY.jpg" alt="TroveLedger riding a golden bull through a festive scene, next to a dancer in traditional Indian clothing" style="max-width:100%;"/>433</details>434 435<details>436<summary>🇬🇧 FTSE 100 (December 22, 2025) – Britain weathers the storm</summary>437 438The **FTSE 100** represents 100 of the most capitalized and liquid firms on the London Stock Exchange, spanning finance, energy, consumer goods, healthcare, and industrial sectors.  439As the UK is no longer part of the European Union, this addition extends TroveLedger’s European coverage beyond the Eurozone without overlap with previously added indices.440 441- [Download symbol list (FTSE.txt)](Symbols/FTSE.txt)442 443<img src="media/FTSE.jpg" alt="TroveLedger safeguarding British market wealth along the Thames during a storm" style="max-width:100%;"/>444</details>445 446<details>447<summary>🇺🇸 S&P 500 (December 19, 2025) – America answers the call</summary>448 449The complete **S&P 500 Index** (503 constituents) has been fully integrated, adding **173 new symbols**.  450This provides the premier US large-cap benchmark with extended intraday histories – ideal for multi-sector trading bot training.451 452- [Download symbol list (SPX.txt)](Symbols/SPX.txt)453 454<img src="media/SPX.jpg" alt="TroveLedger as Uncle Sam proudly presenting the S&P 500 treasure chest" style="max-width:100%;"/>455</details>456 457<details>458<summary>🇭🇰 Hang Seng Index (December 18, 2025) – Asia opens its doors</summary>459 460The **Hang Seng Index (HSI)** adds **82 entirely new symbols** – major Hong Kong-listed companies with strong China exposure across finance, tech, energy, and consumer sectors.461 462- [Download symbol list (HSI.txt)](Symbols/HSI.txt)463 464<img src="media/HSI.jpg" alt="TroveLedger welcoming representatives to the HSI vault" style="max-width:100%;"/>465</details>466 467<details>468<summary>🇪🇺 EURO STOXX 50 (December 17, 2025) – Europe uncovers its treasures</summary>469 470The **EURO STOXX 50** introduces **50 blue-chip companies** from the Eurozone, spanning multiple countries and sectors – a cornerstone for European market exposure.471 472- [Download symbol list (STOXX50.txt)](Symbols/STOXX50.txt)473 474<img src="media/STOXX50.jpg" alt="TroveLedger unveiling the EU flag from a treasure chest labeled STOXX50" style="max-width:100%;"/>475</details>476 477 478## 🔖 Citation479 480If you use TroveLedger in your work, please cite it as:481```482@dataset{Traders-Lab_TroveLedger_2025,483  author = {Traders-Lab},484  title = {TroveLedger Financial Time Series Dataset},485  year = {2025},486  url = {https://huggingface.co/datasets/Traders-Lab/TroveLedger}487}488```489 490### 💰 Support the Treasury Expansion491 492If you're training AI models or building quant pipelines with the TroveLedger dataset, consider supporting further global indices:493 494[![Support on Buy Me a Coffee](https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png)](https://www.buymeacoffee.com/mathiasbacs)495 496Thank you for helping grow the trove! 497 498## 🔚 Final note499 500Markets are not measured by size alone —  501but by how faithfully their records endure.