CoolFace
Datasetpublic

RevolutionCrossroads/nara_revolutionary_war_pension_files

Dataset Card for American Revolutionary War Pension Files Dataset Summary A dataset derived from the Case Files of Pension and Bounty-Land Warrant Applications Based on American Revolutionary War Service, ca. 1800–ca. 1912 (NARA Catalog Series, NAID 300022). This dataset includes page-level records with digitized images, original extracted text (by Family Search), AI-generated OCR, and human-created transcriptions where available. It offers a unique window into… See the full description on the dataset page: https://huggingface.co/datasets/RevolutionCrossroads/nara_revolutionary_war_pension_files.

sourceHugging Facecc0-1.0updated 1mo agoView on Hugging Face
3likes1.2kdownloads
Dataset Card

Dataset Card for American Revolutionary War Pension Files

Table of Contents

Dataset Summary

A dataset derived from the Case Files of Pension and Bounty-Land Warrant Applications Based on American Revolutionary War Service, ca. 1800–ca. 1912 (NARA Catalog Series, NAID 300022). This dataset includes page-level records with digitized images, original extracted text (by Family Search), AI-generated OCR, and human-created transcriptions where available. It offers a unique window into the lives of veterans and their families in the decades following the American Revolutionary War and provides a foundation for research, machine learning, genealogy, and public history projects.

Dataset Description

In honor of the 250th anniversary of the founding of the United States, the Smithsonian Institution prepared this dataset for the Revolution Crossroads initiative using data, metadata, and digital objects publicly available from the National Archives and Records Administration Catalog.

The dataset is derived from the National Archives and Records Administration series Case Files of Pension and Bounty-Land Warrant Applications Based on American Revolutionary War Service, ca. 1800–ca. 1912 (NAID 300022). This series includes more than 82,000 case files containing over 2 million digitized pages.

The pension files preserve personal stories and supporting documents from veterans and their families who applied for pensions or land grants after the American Revolutionary War. These records often include details such as where and when a veteran served, and in which battles they took part. Many files also include applications from widows seeking benefits, offering glimpses into marriages, family life, and community ties.

The dataset includes:

  • —2.2 million page-level records describing digitized pages from pension and bounty-land warrant case files
  • —Metadata fields exported from the National Archives Catalog and converted to Parquet format for analysis
  • —Links to page images, catalog page records, and multi-page PDFs (where available) with persistent identifiers
  • —Extracted text for all records and human-created transcriptions for approximately 39 percent of files (as of May 2026)

This dataset was prepared to support research and experimentation at the intersection of cultural heritage and artificial intelligence. It provides a structured corpus for examining the personal and social history of the Revolutionary era, testing methods for handwriting recognition and transcription, and developing tools for large-scale text analysis, visualization, and discovery. Together, these documents provide a rich and sometimes deeply personal view of life during and after the American Revolutionary War.

Dataset Details

Prepared by: Smithsonian Institution, Office of Digital & Innovation staff Shared by: Revolution Crossroads Language(s): English, some French License: Public domain Data Pull Date: 2026-04-21

Dataset Source

Repository: National Archives Catalog, National Archives and Records Administration Series: Case Files of Pension and Bounty-Land Warrant Applications Based on American Revolutionary War Service, ca. 1800–ca. 1912 National Archives Identifier: 300022 Record Group: Records of the Department of Veterans Affairs, 1773–2007

Curation Rationale

This dataset was prepared as part of the Revolution Crossroads project to enable large-scale analysis and reuse of Revolutionary War pension records in recognition of the 250th anniversary of the United States.

Revolution Crossroads aims to make available a structured dataset that:

  • —Provides page-level access to one of the most important archival series documenting the aftermath of the American Revolutionary War
  • —Brings together images, extracted text, and transcriptions for multi-modal analysis
  • —Supports study of handwriting recognition, transcription validation, and information retrieval at scale
  • —Creates opportunities for public engagement, genealogy, and family history research

Because of these qualities, the dataset lends itself to a range of applications, including:

  • —Exploring the social and personal history of veterans and their families in the early republic
  • —Studying connections between individuals, places, and events documented in the files
  • —Evaluating and improving OCR and transcription methods for handwritten documents
  • —Developing large-scale text retrieval and entity recognition tools for digitized archival materials
  • —Supporting genealogy, public history, and classroom projects related to the American Revolution

Dataset Creation

This dataset was assembled for hosting on Hugging Face by the Smithsonian Institution’s Office of Digital & Innovation staff as part of the Revolution Crossroads project. All records from the National Archives Catalog series Case Files of Pension and Bounty-Land Warrant Applications Based on American Revolutionary War Service, ca. 1800–ca. 1912 (NAID 300022) were included.

The dataset was prepared by pulling JSON-formatted data from three NARA API endpoints – one for the record metadata, one for the extracted text, and one for the transcriptions (where available). These sources were combined and flattened so that each page is represented as a single record, and then uploaded to Hugging Face for efficient analysis and hosting, which includes automatic conversion to the Parquet format.

Data Collection and Processing

Processing Steps

  • —Retrieved JSON-formatted data from three NARA API endpoints (records, extracted text, transcriptions)
  • —Combined datasets into a single table keyed by record identifiers
  • —Flattened into one record per line, with page-level granularity
  • —Removed pages from microfilm target sheet only files (about 18,000 pages)
  • —Removed scanned microfilm reels which have not yet been broken down by Pensioner. (approximately 70,000 pages)

Supporting Files Available

JSON-formatted data from the three NARA API endpoints are available in the Files tab. Supporting files are organized into NAID-based ranges, with each range corresponding to a chunk of records and its associated data.

  • —records/ contains JSON output from the NARA records API. Files are named by NAID range. Example: naid_111410768-111420673.json
  • —extracted_text/ contains JSON output for extracted text associated with the same NAID ranges. Example: naid_111410768-111420673_ocr.json
  • —transcriptions/ contains JSON output for transcriptions, where available, associated with the same NAID ranges. Example: naid_111410768-111420673_transcriptions.json

Accessing Page-Level Images

Due to repository file limits, the full set of page-level images (over 2 million images) is not included directly in this dataset as supplementary files. Instead, each row includes a pageImageURL field pointing directly to the corresponding image.

These URLs can be:

  • —opened directly in a browser for one-off access
  • —used in scripts or command-line tools for batch downloading
Downloading Images in Bulk (CLI)

To download multiple images, create a text file containing one image URL per line (for example, exported from the pageImageURL column in the parquet file), then use standard tools like wget or curl.

Example with wget:

bash
wget -i image_urls.txt

Example with curl:

bash
xargs -n 1 curl -O < image_urls.txt
High-Throughput Downloading (S3 / Advanced)

For large-scale workflows, it may be more efficient to download directly from Amazon S3 using parallelized tools.

The pageImageURL values point to files stored in S3 and can be converted to S3 paths by replacing:

https://s3.amazonaws.com/

with:

s3://

For example:

https://s3.amazonaws.com/bucket/path/file.jpg becomes: s3://bucket/path/file.jpg

To use high-performance tools such as s5cmd, create a command file with one copy command per line:

bash
cp s3://bucket/path/image1.jpg ./images/image1.jpg
cp s3://bucket/path/image2.jpg ./images/image2.jpg

Then run:

bash
s5cmd --no-sign-request run s5cmd_commands.txt

The --no-sign-request flag is used because the source files are publicly accessible. Tools like s5cmd also support parallel downloads and configurable worker counts for improved performance.

Notes
  • —For small or exploratory use, direct URLs (pageImageURL) are the simplest option
  • —For large-scale downloads, S3-based tools provide significantly better performance
  • —Users can filter the parquet file (e.g., by NAID) before generating URL or S3 path lists to target specific records
  • —For very large jobs, users may wish to tune concurrency, retries, or logging settings depending on their environment
  • —When downloading very large numbers of files, it may be helpful to organize outputs into subdirectories (e.g., by ID range or record identifier), as many file systems perform poorly with extremely large numbers of files in a single directory.

Accessing a Specific Data Version

This dataset may be updated over time as source data is revised, normalized, or restructured. If you are using the data for a project, we recommend accessing a specific tagged version so your workflow remains stable and reproducible.

You can access a tagged version using the revision parameter in load_dataset:

python
from datasets import load_dataset
import polars as pl

dataset_name = "RevolutionCrossroads/nara_revolutionary_war_pension_files"
commit_tag = "Source_2026-04-21"

ds = load_dataset(
    dataset_name,
    revision=commit_tag,
    split="train"
)

# Convert to pandas
pandas_df = ds.data.table.to_pandas()

# Convert to Polars
polars_df = pl.from_arrow(ds.data.table)

This example loads the specified tagged dataset version and converts it to either a pandas or Polars dataframe for analysis.

Available Data Versions

Using a tagged version is recommended for research, publication, or project workflows where consistency over time is important.

AI OCR Enrichment

This dataset has been enriched with AI-generated OCR produced using DataLab's Chandra OCR 2, a Vision Language Model (VLM) for document understanding.

Complete Chandra OCR 2 processing outputs are stored in the Revolution Crossroads VLM OCR Hugging Face bucket. The bucket includes a README describing the processing pipeline, output formats, and directory structure.

One additional field has been added:

  • —RevX_OCR_text contains plain text extracted from the page using Chandra OCR 2. This field supplements the original extracted text (fs_extractedText) and is provided to support downstream analysis and comparison of OCR methods.

These outputs represent the direct results of automated AI processing. They have not been manually validated or corrected and should be treated as machine-generated data.

Not every record in this dataset currently includes Chandra OCR 2 outputs. Initial processing was completed in March 2026 and focused on pages that had not yet been human transcribed. Since then, the source collections have been refreshed with additional materials and new transcriptions. A second round of Chandra OCR 2 processing is currently underway, and additional processing outputs will be incorporated into future dataset releases.

Quality Considerations

  • —The Hugging Face dataset includes all records from the Catalog export; no records were added or removed
  • —Extracted text is present for all records, produced through automated processes including AI assistance (via FamilySearch partnership)
  • —Human transcriptions were created through the NARA Citizen Archivist program; these currently cover approximately 39 percent of files (as of May 2026)
  • —Metadata reflects the Catalog at the time of export and is updated by NARA on an ongoing basis

Dataset Structure

Record Level Each record in the dataset corresponds to a single digitized page from a pension file.

Identifier Field Types Identifier fields such as NAID, pdfObjectId, and pageObjectId are stored as strings in this release. During dataset preparation, integer representations caused some interfaces, including the Hugging Face dataset viewer, to display large numeric values with inserted commas for readability. Users performing numeric operations or joins may find it useful to cast these fields to integer types within their analysis environment.

Data Fields

Core Identifiers

  • —NAID (large_string) National Archives Identifier for the pension file. Example: 111769430
  • —naraURL (large_string) Link to the pension file record in the National Archives Catalog. Example: https://catalog.archives.gov/id/111769430
  • —title (large_string) Title of the pension or bounty land warrant application file, often including the veteran’s name and case type. Example: Revolutionary War Pension and Bounty Land Warrant Application File S 33948, Wm Woodbury, Mass. Note on File Title Abbreviations
  • —S — Survivor / Soldier
  • —R — Rejected
  • —W — Widow
  • —B, BLW, B L Wt. — Bounty Land Warrant
  • —OW — Old War
  • —NA Acc — National Archives Accession number (placeholder when no pension file was located)

Notes on errors: Some letters (P, K, T, H, M) are mistranscriptions of the above. These are being corrected in the NARA Catalog but may still appear in this dataset until a future data pull.

  • —levelOfDescription (large_string) Archival level assigned to the record in the National Archives Catalog. Most records in this dataset are described at the fileUnit level. A small number are described as item records, typically illustrations or fraktur-style artworks that were cataloged separately by NARA. Example: fileUnit

Dates and Identifiers

  • —logicalDate (large_string) Normalized or machine-readable date, if available. Example: 1831-06-20
  • —variantControlNumbers (large_string) Alternate identifiers linked to the file. Example: [{"number": "Fold3 2018", "type": "Search Identifier"}]

File-Level Data

  • —Note: pdfObjectId and pdfURL refer to a multi-page file that includes all pages for a given pension application. These values repeat across all pages in the same file.
  • —pdfObjectId (large_string) Identifier for the file-level PDF. Example: 19850379
  • —pdfURL (large_string) URL to the full multi-page PDF. Example: https://s3.amazonaws.com/NARAprodstorage/lz/rediscovery/14974.pdf

Page-Level Data

  • —pageObjectId (large_string) Unique identifier for a single page image. Example: 19850379
  • —pageNumber (int64) Page order within the pension application file. Example: 11
  • —pageLink (large_string) URL to the specific page within the National Archives Catalog viewer. This provides page context within the larger file and is not the direct image URL. Example: https://catalog.archives.gov/id/196380975?objectPage=11
  • —pageImageURL (large_string) Direct URL to the page image file stored in Amazon S3. Example: https://s3.amazonaws.com/.../161779_00885.jpg
  • —pageImageType (large_string) File format of the page image. Example: JPG
  • —extractedNames (large_string) Indexed names associated with the page, when available. During processing, non-name text such as “Image provided by Fold3” was removed from this field. Example: Names found on this page include: Joseph Perkins.

Extracted Text (OCR)

  • —fs_extractedTextID (large_string) Identifier for the extracted text object. Example: 33948 Revy INVALID. File No. 33948 Index William Woodbury Sir Rev War Act: 18 March 18 ...
  • —fs_extractedText (large_string) Extracted text created via OCR/AI processes. Present for all records. Example excerpt: “State of Pennsylvania, County of Chester, On this 5th day of March 1833 personally appeared before the court John Smith, aged seventy-eight years…”
  • —fs_extractedTextDate (large_string) Date OCR text was uploaded. Example: 2024-11-20T20:32:03.000Z
  • —RevX_OCR_text (large_string) Plain text extracted from the page using DataLab's Chandra OCR 2 Vision Language Model (VLM). This field supplements the original extracted text (fs_extractedText) and may be null if the page has not yet been processed. Example excerpt: "Revolutionary War Pension File S. 14,123..."

Transcriptions (Citizen Archivists)

  • —transcriptionIngestDate (timestamp[ns]) Date the transcription was ingested into this dataset. If multiple transcriptions were present for a page, the latest ingest date was used. Example: 2026-04-22T13:08:21.078000
  • —transcriptionID (large_string) Identifier for the transcription record, if one exists. Example: null or a unique string for transcribed pages.
  • —transcriptionText (large_string) Human-created transcription text. Available for approximately 39 percent of files. Example excerpt: “That he enlisted in the month of April 1777 for the term of three years…”

Source Data

Series of Records

Case Files of Pension and Bounty-Land Warrant Applications Based on American Revolutionary War Service, ca. 1800–ca. 1912 (NAID 300022)

Record Group

Records of the Department of Veterans Affairs, 1773–2007

Microfilm Publications

M804 — Revolutionary War Pension and Bounty-Land Warrant Application Files

Coverage

This series documents the time period ca. 1775–ca. 1900

Arrangement

Arranged alphabetically by last name of veteran (soldier)

Record Creators

Department of the Interior. Bureau of Pensions (1849–1930) (most recent) War Department. Office of the Secretary (1789–1947) (predecessor) War Department. Military Bounty Lands and Pension Branch (ca. 1810–1815) (predecessor)

Custody History

This series was maintained by the Office of the Secretary of War until ca. 1810, by the Military Bounty Lands and Pensions Branch from ca. 1810–1815, and thereafter by the Bureau of Pensions. The records are currently held by the National Archives and Records Administration.

Known Gaps

There is a gap of 84 rolls of images missing from this series. These are currently being digitized by the National Archives and will be added to the Catalog upon completion.

Digitization Partnerships

The National Archives microfilm was originally digitized by Fold3 through a partnership agreement.

In April 2024, the Archives partnered with FamilySearch to run all images through an AI large language model to produce first-draft extracted text. These drafts were added to the images in the National Archives Catalog as partner-contributed text.

Volunteer Crowdsourcing

National Archives volunteers, called Citizen Archivists, have been copying extracted text into the Catalog transcription module, reviewing and cleaning up errors, and publishing the corrected versions. These human-created transcriptions are available for approximately 39 percent of files to date (May 2026).

Personal and Sensitive Information

No known personal or sensitive information is included. Users should be aware that records may contain outdated terminology reflecting the period in which they were created.

Considerations for Using the Data

Risks and Limitations

This dataset represents a large-scale export of pension file pages from the National Archives Catalog. It is not a comprehensive or definitive edition of the underlying archival series.

  • —Some pension files are incomplete due to missing rolls of microfilm
  • —Extracted text accuracy varies; errors are especially common for handwritten pages
  • —Human transcriptions improve accuracy but are only available for about 39 percent of files
  • —Names, places, and dates may appear with inconsistent spelling or formatting
  • —Cataloging metadata is updated by NARA on an ongoing basis

Recommendations

Users should be aware of the limitations described above when using the dataset. It is recommended that researchers consult the National Archives Catalog for the most up-to-date versions of the records. Researchers should validate extracted information against the underlying page images whenever possible. When comparing OCR performance or conducting downstream analyses, users should distinguish between the original extracted text (fsextractedText), AI-generated Chandra OCR 2 (RevXOCR_text), and human-created transcriptions (transcriptionText), where available.

Additional Information

Citation Information

BibTeX

@dataset{RevolutionCrossroads_NARA_2025, author = {Revolution Crossroads Project Team}, title = {American Revolutionary War Pension Files}, year = {2025}, publisher = {Hugging Face}, url = {https://huggingface.co/datasets/RevolutionCrossroads/nara_revolutionary_war_pension_files}, doi = {10.57967/hf/6529} }

APA Revolution Crossroads Project Team. (2025). American Revolutionary War Pension Files [Data set]. Hugging Face. https://doi.org/10.57967/hf/6529

Glossary

  • —NAID: National Archives Identifier, a unique reference number for archival series and files
  • —Extracted text: Machine-generated text created from scanned images using optical character recognition or AI-assisted methods. Present for all records in this dataset
  • —Transcription: Human-created text entered manually by volunteers or staff. Available for approximately 27 percent of files
  • —Fold3: Commercial partner that digitized NARA microfilm
  • —FamilySearch: Partner of the National Archives that created extracted text for the series
  • —Citizen Archivist: NARA’s volunteer program for transcription and metadata improvement

Dataset Card Contact

revolutioncrossroads@si.edu