MatthewAIExplorer/second-circuit-decisions-2020-2025

Second Circuit Decision Search Records, 2020–2025

This repository is a reproducible research collection of opinions and summary orders made discoverable by the United States Court of Appeals for the Second Circuit. Inclusion is based on the result row's displayed Date field, whose precise filing, publication, or indexing semantics are not defined by the search interface. The collection filter covers those search-result dates from January 1, 2020 through December 31, 2025, inclusive. No case-type or disposition filter was applied; that means the query was unfiltered, not that every case type or outcome is represented, exhaustively covered, or labeled.

This is not an official court archive and does not represent every appeal filed in the Second Circuit. It contains index records and decision documents returned by the court's public decision search during the collection snapshot. This Hugging Face release publishes the research collection, not a trained model; no model was trained as part of this phase.

Source

• —Court search page: Second Circuit Decisions
• —Verified search endpoint: https://ww3.ca2.uscourts.gov/dtSearch/dtisapi6.dll
• —Document files: direct PDF links returned by the court search
• —Discovery snapshot represented in the local manifest: September 8, 2026 UTC

The active full-range discovery found 7,589 distinct index rows and source PDF URLs. Counts below are grouped by the court's search-result date; they are search-result records, not a claim about unique legal matters, all appeals, or all activity in the court.

The hosted reports/validation_report.md and reports/pilot_validation.md files summarize download, extraction, duplicate, OCR, and coverage checks. A local collector run also generates detailed machine-readable reports and operational logs; those local-state files are intentionally excluded from the public release.

Release snapshot

• —Release: v1.0.0
• —Publisher: MatthewAIExplorer
• —Discovery snapshot: September 8, 2026 UTC
• —Records: 7,589; all have hash-verified PDFs and page-bounded extracted text in the local validation run
• —Strict validation: 0 critical issues and 3 retained noncritical warnings—two OCR attempts with no improvement and one readable PDF parser-repair warning
• —Coverage reconciliation: 7,589 reported search rows, 7,589 discovered records, and 0 identified gaps in the completed partition tree
• —PDF-face date comparison: 7,387 unambiguous candidates, including 7,313 exact matches and 74 differences from the search-result date; 201 had no candidate and 1 was ambiguous; 7 PDF-face candidates fall outside the search-result-date window
• —Optional source-supported metadata: case_type is null for all 7,589 records; precedential_status is explicitly nonprecedential for 5,745 records and null for 1,844. Null does not mean precedential.

Canonical export checksums:

Hosted release layout

data/
  second_circuit_decisions.parquet
alternate_formats/
  second_circuit_decisions.jsonl
  second_circuit_inventory.csv
scripts/
  collect.py                    command-line entry point
  ca2_pipeline.py               discovery, download, validation, and extraction
  validate.py                   collection and coverage audit
  export.py                     deterministic dataset exports
tests/                           offline parser and invariant tests
reports/
  pilot_validation.md
  validation_report.md
LICENSE                          rights and licensing notice
SHA256SUMS                       canonical export checksums
requirements.txt

A working collector checkout additionally contains the unchanged source PDFs under raw_pdfs/YYYY/, separate page-bounded text under text/YYYY/, the resumable data/manifest.sqlite, and detailed files under logs/. Those operational artifacts are retained locally and are not part of the compact hosted release. The Parquet and JSONL exports include the extracted full text, source URLs, hashes, provenance, and quality fields needed to audit each hosted record.

Original PDFs are never rewritten. OCR is applied only while producing separate text output. Byte-identical downloads reuse the canonical stored PDF path; the manifest keeps every source observation and its duplicate relationship. Amended or reissued records with distinct source identity remain separate versions.

Collection method

The collector uses the court's actual dtSearch form contract rather than constructing unverified PDF URLs:

1. 1.It sends a form-encoded POST to the verified dtSearch endpoint with request=*, searchType=allwords, cmd=search, SearchForm=%%SearchForm%%, OrigSearchForm=/decisions.html, pageSize=25, sort=date, and an inclusive xfilter(date "YYYY/MM/DD~~YYYY/MM/DD") condition.
2. 2.It searches the court's Opinion and Summary Order indexes. The verified index values are *{aa12e167958cdbcaa709fa14b9161a4a} OPN and *{aad0964f04f3e9c420e057fd415efe0c} SUM.
3. 3.The endpoint can emit headerless HTTP/0.9-style responses. Discovery therefore invokes curl with both --http1.1 and --http0.9. Pagination replays every hidden NextPageForm field verbatim and retains the court's 25-result page size.
4. 4.The court search may cap large result sets near 750 rows. The collector recursively bisects inclusive date ranges whenever a query reports more than 700 results. A capped one-day combined query is split by document index. Any irreducible or incomplete leaf becomes a recorded coverage gap.
5. 5.Result rows are parsed from table.ResultsTable. The caption comes from the row's captionStr JavaScript value after JavaScript and HTML entity decoding. Docket strings, displayed date, displayed type, and the PDF link are retained as source metadata.
6. 6.Network requests are serial and paced by at least 0.75 seconds by default. Temporary failures use capped exponential backoff with jitter and up to five attempts. Access-denial responses stop the run and remain recorded for later diagnosis.
7. 7.PDFs must have a PDF signature, nontrivial size, a readable page tree, and at least one page. SHA-256 is calculated over the unchanged response bytes. A missing %%EOF marker is retained as a validation warning.
8. 8.A deterministic, approximately balanced 100-document pilot spans year and document-type strata. The full run starts only after every pilot PDF and page-boundary invariant passes.
9. 9.Native text is extracted page by page with pypdf. Pages are joined with one form-feed character (\f) and no trailing separator. Sparse or unreadable pages are flagged for OCR. When pdftoppm and Tesseract are available, only flagged pages are rendered and OCRed; native text is retained when it is better.

The SQLite partitions and observations tables preserve reported search counts, result rows, pagination completion, and the active root partition. Reconciliation sums only completed leaf descendants of that active root, preventing an overlapping smoke query or earlier run from inflating coverage totals.

Installation

Python 3.11 or newer is recommended. The current collection was tested with Python 3.13. To reproduce or extend the collection from the hosted release:

git clone https://huggingface.co/datasets/MatthewAIExplorer/second-circuit-decisions-2020-2025
cd second-circuit-decisions-2020-2025
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

Required external command:

command -v curl

OCR is optional but recommended. The collector records ocr_status=unavailable instead of altering or hiding a document when these commands are absent.

command -v pdftoppm
command -v tesseract

On macOS with Homebrew, install the optional OCR tools with:

brew install poppler tesseract

Run, resume, validate, and export

Run the 100-document pilot and automatically continue through the full date range after the pilot gate passes. The commands below run from the cloned repository root:

source .venv/bin/activate
python scripts/collect.py \
  --start-date 2020-01-01 \
  --end-date 2025-12-31 \
  --pilot-size 100 \
  --auto-continue

Resume safely after interruption. This same command revisits incomplete discovery partitions, skips already valid artifacts, retries unfinished records, reruns the pilot gate, and continues the full collection:

cd ~/Desktop/second-circuit-dataset
source .venv/bin/activate
python scripts/collect.py \
  --start-date 2020-01-01 \
  --end-date 2025-12-31 \
  --pilot-size 100 \
  --auto-continue \
  --resume

When discovery is already reconciled and only document processing remains, --skip-discovery avoids another search request:

python scripts/collect.py \
  --start-date 2020-01-01 \
  --end-date 2025-12-31 \
  --pilot-size 100 \
  --auto-continue \
  --resume \
  --skip-discovery

Generate the full validation report. --strict returns a nonzero exit status for corrupt/missing artifacts, unfinished records, invalid metadata values, page-boundary violations, or coverage gaps. Source metadata that the court does not provide remains reported as null rather than fabricated.

python scripts/validate.py \
  --start-date 2020-01-01 \
  --end-date 2025-12-31 \
  --strict

Create the three deterministic exports. Strict export stops before replacing output files unless every manifest record in the requested date range has a hash-verified, parseable PDF and valid page-bounded UTF-8 text, and the completed search partitions reconcile without coverage gaps.

python scripts/export.py --strict

Run the offline parser and pipeline tests:

python -m pytest -q

Export formats

• —second_circuit_decisions.jsonl: one UTF-8 JSON object per retained document version, including full_text.
• —second_circuit_decisions.parquet: the same rows and full text with an explicit PyArrow schema and Zstandard compression.
• —second_circuit_inventory.csv: the same inventory fields except full_text. List fields are compact JSON arrays (or JSON null when unknown) so docket numbers and flags remain lossless.

The Hugging Face Dataset Viewer and default loader are configured to use only the Parquet export as the single full split. full denotes the complete corpus, not a training or evaluation designation. The JSONL and CSV files under alternate_formats/ are alternate representations and are not additional observations.

from datasets import load_dataset

decisions = load_dataset(
    "MatthewAIExplorer/second-circuit-decisions-2020-2025",
    split="full",
)

Exports sort deterministically by decision date, docket string, case name, and document ID. Every logical record is retained by default, including rows whose PDF bytes match another row; duplicate_of and the shared hash preserve that relationship. Pass --exclude-byte-duplicates only for an explicitly reduced view. Explicit amended/reissued versions are always retained.

Dataset schema

Null means the value is unknown or unsupported by the source. The exporter does not infer case type, precedential status, outcome, or disposition.

decision_date is a deprecated compatibility alias, not an independently verified legal decision date. New consumers should use search_result_date and consult pdf_decision_date, date_mismatch, and metadata_evidence_json as separate evidence. PDF-face extraction inspects the first physical page; candidates on later pages may remain null, and a stated-term formula is not literally a Decided: label.

Validation and reconciliation

In a working collector checkout, scripts/validate.py opens the local manifest read-only and checks:

• —manifest, collected, and processing-status counts;
• —counts by court search-result year and document type;
• —SHA-256 duplicate groups and collector duplicate links;
• —missing court metadata, with source-unsupported fields reported separately;
• —HTTP/download failures and unfinished manifest records;
• —PDF signature, size, stored hash, parser readability, encryption, and page count;
• —UTF-8 text presence, extraction/OCR status, and text character counts;
• —exactly page_count - 1 form-feed separators;
• —JSON syntax, hashes, timestamps, source URLs, page-count vectors, OCR-state consistency, and source-evidence invariants;
• —noncritical comparisons between the court search-result date and unambiguous PDF-face dates;
• —active-root leaf partition totals, reported-versus-discovered counts, capped/failed partitions, and uncovered date intervals.

The local JSON report contains complete structured issue lists. The hosted Markdown report gives the main totals and bounded review tables. scripts/export.py separately reads all three outputs back, verifies row counts plus ID sequence/order and basic schema across JSONL, Parquet, and CSV, and records local output sizes and SHA-256 values.

Court text and collector annotations

Source material is kept logically separate from collector-created transcriptions and annotations:

• —Court-source artifacts: unchanged local PDFs; the court search row's caption, raw docket string, displayed date and type; and source URLs.
• —Derived transcription and metadata: text/ and full_text, OCR output, normalized docket arrays, PDF-face date candidates, generated identifiers, document-family/version/duplicate links, paths, timestamps, hashes, extraction methods, OCR statuses, and quality flags.

OCR text is a derived transcription and can differ from the visual PDF. Use the PDF as the authoritative document and the metadata-evidence fields to audit normalization decisions.

Intended use and sensitive information

The collection is intended for legal research, corpus analysis, information retrieval, reproducibility work, and study of published appellate decision documents. It is not intended for legal advice, automated eligibility or adverse decisions, profiling named people, re-identification, or unreviewed outcome prediction.

Opinions and summary orders can contain names and sensitive allegations or facts involving criminal proceedings, immigration, health, finances, employment, family circumstances, and other legal matters. Although these details appear in public judicial records, aggregation can increase discoverability and potential harm. No independent de-identification or exhaustive personally identifying information audit was performed. Users should apply data minimization, access controls where appropriate, human review, and heightened care before redistributing excerpts or linking records to individuals.

For a correction or removal concern about this hosted compilation, open a Discussion in the Hugging Face repository with the document ID and source URL. The court remains the authoritative source, and a change to this compilation cannot remove or modify the court's source record.

Limitations

• —Coverage is limited to documents returned by the court's public decision search at the snapshot time. Sealed material, documents absent from the index, later corrections, later backfills, and other kinds of orders may be missing.
• —A decision collection is not a population of all appeals. Appeals without a returned opinion or summary order are outside this dataset.
• —Search-result counts and records can change after the snapshot. The active partition tree and validation report capture what was reconciled during this run.
• —The collection window and year tables use search_result_date (also retained as the compatibility field decision_date), not pdf_decision_date. A true date_mismatch records disagreement between those source values; it does not declare either date erroneous or silently correct the court record.
• —pdf_decision_date is automated PDF-face evidence, not a date verified against docket chronology. PDF faces can contain typos, amended or reissued documents can retain the original decision date, and complete/bundled PDFs can make the governing date ambiguous; verify consequential chronology against the PDF and docket.
• —Captions and docket strings can be irregular. Parsed docket arrays should be checked against docket_number_raw and the PDF for consequential use.
• —Version relationships use explicit filename markers plus matching normalized docket sets and document type. They are research metadata, not a court-certified procedural history.
• —Native PDF extraction can lose reading order, footnotes, tables, or typography. OCR introduces recognition errors. Quality flags identify likely problems but do not guarantee correctness.
• —The collection reproduces public-record text without adding redaction. Search indexing, extraction, and aggregation can make sensitive information easier to discover than in the source PDFs.
• —There are no verified outcome labels in this phase. The collector applies no outcome filter but does not infer dispositions. Outcome prediction would require separately collected pre-decision materials, a leakage review, and verified labels.

Rights and responsible use

The source documents are public federal judicial records. Federal judicial opinions are generally treated as uncopyrightable public law, but this dataset does not assert a blanket license over every element of every PDF or over the court website. Filings quoted or reproduced in decisions, photographs, appendices, seals, and other third-party material can have separate rights or use restrictions. Review the court's current terms and the relevant law before redistribution.

The Hugging Face metadata therefore uses license: other; see LICENSE for the element-by-element rights notice. Preserve source attribution and document URLs, do not present derived text as an official court copy, and verify consequential legal facts against the original PDF and current docket. This collection is for research and is not legal advice.

Citation

@dataset{matthew_ai_explorer_second_circuit_2026,
  author    = {MatthewAIExplorer},
  title     = {Second Circuit Decision Search Records, 2020--2025},
  year      = {2026},
  version   = {1.0.0},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/datasets/MatthewAIExplorer/second-circuit-decisions-2020-2025}
}