CoolFace
Datasetpublic

datamatters24/research-document-archive

Research Document Archive Computational analysis of 234,630 declassified U.S. government documents across 7 archival collections. Output of a 13-step ML pipeline extracting OCR text, entities, topics, keywords, redactions, and semantic embeddings from 3.1 million pages. Files File Rows Description documents.parquet 234,630 Document metadata: id, source_section, file_path, file_hash, total_pages pages/<section>.parquet 3.1M Per-page OCR text + 1536-dim… See the full description on the dataset page: https://huggingface.co/datasets/datamatters24/research-document-archive.

sourceHugging Facecc0-1.0updated 5mo agoView on Hugging Face
0likes59downloads
Dataset Card

Research Document Archive

Computational analysis of 234,630 declassified U.S. government documents across 7 archival collections. Output of a 13-step ML pipeline extracting OCR text, entities, topics, keywords, redactions, and semantic embeddings from 3.1 million pages.

Files

FileRowsDescription
documents.parquet234,630Document metadata: id, sourcesection, filepath, filehash, totalpages
pages/<section>.parquet3.1MPer-page OCR text + 1536-dim sentence-transformer embeddings. Sharded by collection.
entities/<section>.parquet31MNamed entities (spaCy en_core_web_lg): PERSON, ORG, DATE, GPE, FAC, LOC, NORP, EVENT. Sharded by collection.
document_topics.parquet234,629BART-large-MNLI zero-shot topic assignments (per doc, top-1 topic + probability)
document_features.parquet1.17MEAV feature table: redaction_summary, forensic_metadata, bertopic, sentiment, topic_distribution, exact_duplicate
document_keywords.parquet3.5MTF-IDF keywords (top 15 per document, unigrams + bigrams)
document_dates.parquet234,630Inferred document dates (regex + header parsing)
document_events.parquet296KDocument-to-historical-event correlations (20 crisis events)
historical_events.parquet20Crisis event dictionary (event_name, date range, category, keywords)
entity_relationships.parquet2.88MEntity co-occurrence pairs with counts, distances, and sample documents

Collections

source_sectionDocumentsSource
cia_declassified1,605CIA Reading Room
cia_mkultra1,936MKULTRA release
cia_stargate13,937Stargate remote viewing program
doj_disclosures—DOJ public disclosures
house_resolutions181,092House.gov bill text (GovInfo API)
jfk_assassination35,979National Archives JFK release
lincoln_archives21Library of Congress

Loading

python
from datasets import load_dataset
docs = load_dataset("datamatters24/research-document-archive", data_files="documents.parquet")

# Or load a specific sharded table:
import pyarrow.parquet as pq
pages = pq.read_table("hf://datasets/datamatters24/research-document-archive/pages/cia_mkultra.parquet")

Methodology

  • —OCR: Tesseract + PyMuPDF
  • —NER: spaCy en_core_web_lg
  • —Topics: BART-large-MNLI (zero-shot) + BERTopic (unsupervised)
  • —Embeddings: sentence-transformers/all-MiniLM-L6-v2 (384-dim) and OpenAI text-embedding-3-small (1536-dim) per page
  • —Redaction detection: OpenCV contour analysis on PDF-rendered pages
  • —Entity relationships: page-window co-occurrence with distance weighting

Code: https://github.com/tedrubin80/Massivedata-Pull