aaron-freedman/banking-concentration-data

Banking Concentration Data (1959–2025)

Public portion of the source data for the banking-concentration visualizations project at https://github.com/aaron-freedman/banking-concentration. This dataset assembles regulatory filings from three US bank and thrift supervisors covering 1959–2025 into a single, loadable collection of parquet and comma-separated values (CSV) files.

Important: three additional files are needed by the full preprocessing pipeline but are not included here because they come from Wharton Research Data Services (WRDS) under a subscription agreement that restricts public redistribution. Researchers with WRDS access can regenerate them using the helper script in the code repository. Those three files ultimately derive from publicly-available Federal Financial Institutions Examination Council (FFIEC) National Information Center (NIC) data, which may eventually be re-sourced from FFIEC bulk downloads directly.

Files on this dataset

Usage

Download everything

pip install huggingface_hub
huggingface-cli download aaron-freedman/banking-concentration-data \
    --local-dir ./raw --repo-type dataset

Load a single file without downloading

import pandas as pd

df = pd.read_parquet(
    "hf://datasets/aaron-freedman/banking-concentration-data/call_reports_bs.parquet"
)
print(df.shape, df.columns.tolist()[:10])

Regenerate the WRDS-sourced files

If you have a WRDS subscription, clone the code repository and run:

git clone https://github.com/aaron-freedman/banking-concentration
cd banking-concentration
pip install wrds
python3 scripts/fetch_wrds_data.py --username YOUR_WRDS_USERNAME

This writes bhc_relationships_1965_2024.csv, entity_attributes.csv, and bank_mergers_1965_2024.csv into ../call-reports/data_CLV/ relative to the repository, matching the paths expected by preprocess.py.

Data sources and attribution

Commercial-bank balance sheets and income statements (1959–2025)

Files: call_reports_bs.parquet, call_reports_is.parquet, historical_call_data_dictionary.xlsx, CLV_README.html.

Original source: Federal Financial Institutions Examination Council (FFIEC) Call Reports — the quarterly regulatory reports of condition and income filed by every US commercial bank with federal bank supervisors. Forms include FFIEC 010, 011, 012, 030, 031, 032, 033, 034 (the set varies over time by reporter and era).

Cleaned and redistributed by: Federal Reserve Bank of New York, under the dataset Balance Sheets and Income Statements of Commercial Banks: 1959 through 2025 (https://www.newyorkfed.org/research/banking_research/balance-sheets-income-statements). The processing pipeline was written by Sergio Correia, Stephan Luck, and Emil Verner (together, "CLV") as part of their research on bank failures.

Please cite the accompanying paper when using these files:

Correia, S. A., Luck, S., & Verner, E. (2026). Failing Banks. The Quarterly Journal of Economics, 141(1), 147–204. https://doi.org/10.1093/qje/qjaf044

Redistribution notice (required by the New York Fed Terms of Use at https://www.newyorkfed.org/privacy/termsofuse):

© Federal Reserve Bank of New York. Content from the New York Fed is subject to the Terms of Use at newyorkfed.org.

Thrift (savings institution) data, 1984–present

Files: thrift_financials.parquet, thrift_history.csv, thrift_institutions.csv.

Original source: Federal Deposit Insurance Corporation (FDIC) — BankFind API and Research & Analysis bulk download files at https://api.fdic.gov/banks and https://www.fdic.gov/resources/data-tools/. US federal public-domain data. No authentication is needed to retrieve these files from FDIC.

thrift_financials.parquet has column names harmonized with the CLV balance-sheet style (lowercase, underscored) so the two files can be concatenated; the underlying data is raw FDIC.

Thrift (savings institution) data, 1967–1989

File: boss_thrifts.parquet.

Original source: Federal Home Loan Bank Board (FHLBB) regulatory filings by Federal Savings and Loan Insurance Corporation (FSLIC)-insured thrifts. FHLBB was the federal thrift regulator until it was abolished by the Financial Institutions Reform, Recovery, and Enforcement Act of 1989 (FIRREA); thrift supervision then moved to the Office of Thrift Supervision (OTS).

Archived at: National Archives and Records Administration (NARA) Record Group 195 (Records of the Federal Home Loan Bank Board), series 635816 — the Branch Office Survey System (BOSS). Publicly downloadable from s3.amazonaws.com/NARAprodstorage/lz/electronic-records/rg-195/BOSS/. US federal public-domain data.

Helper lookups

File: boss_fips_lookup.csv — author-compiled crosswalk from US city/state strings to 5-digit US Census Federal Information Processing Standards (FIPS) county codes, used by the BOSS parser. Underlying Census geographic identifiers are US federal public-domain.

Modifications

No modifications have been made to the underlying data. Columns in thrift_financials.parquet have been renamed from FDIC's native all-caps field names to a lowercase underscored convention that matches the CLV balance-sheet file; no values have been altered. The BOSS parquet is derived from the original 38 fixed-width ASCII files according to the record layouts documented in nara-boss/CLAUDE.md in the code repository.

License

• —Files originating from the Federal Reserve Bank of New York carry the New York Fed Terms of Use (attribution required; public redistribution permitted).
• —Files originating from FDIC, NARA, and the US Census are US federal public-domain.
• —No additional license is applied by this Hugging Face dataset.

See the code repository for related source code, released under the MIT License.