CoolFace
Datasetpublic

BEE-spoke-data/govdocs1-pdf-source

govdocs1: source PDF files [!NOTE] Converted versions of other document types (word, txt, etc) are available in this repo This is ~220,000 open-access PDF documents (about 6.6M pages) from the dataset govdocs1. It wants to be OCR'd. Uploaded as tar file pieces of ~10 GiB each due to size/file count limits with an index.csv covering details 5,000 randomly sampled PDFs are available unarchived in sample/. Hugging Face supports previewing these in-browser, for example this one… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-pdf-source.

sourceHugging Facecc-by-4.0updated 9mo agoView on Hugging Face
6likes4.3kdownloads
Dataset Card

govdocs1: source PDF files

[!NOTE] Converted versions of other document types (word, txt, etc) are available in this repo

This is ~220,000 open-access PDF documents (about 6.6M pages) from the dataset govdocs1. It wants to be OCR'd.

  • —Uploaded as tar file pieces of ~10 GiB each due to size/file count limits with an index.csv covering details
  • —5,000 randomly sampled PDFs are available unarchived in sample/. Hugging Face supports previewing these in-browser, for example this one

Recovering the data

Download the data/ directory (with huggingface-cli download or similar) extract the tar pieces:

sh
cat data_pdfs_part.tar.* | tar -xf - && rm data_pdfs_part.tar.*

processing details

duplicates

exact duplicate PDFs were removed with jdupes. See the log file for details.

By the numbers

Based on the index.csv

Dataset Overview

MetricValuePercentage
Total Documents229,917100%
Successfully Processed229,82499.96%
Broken/Corrupted930.04%
Unique Filenames229,917100%

Document Structure

Page Count Distribution
PagesCountPercentage
2 pages21,8879.5%
1 page19,2828.4%
4 pages14,6406.4%
3 pages12,8615.6%
6 pages9,7704.3%
StatisticValue
Range1 - 3,200 pages
Mean27.8 pages
Median10 pages
Standard Deviation67.9 pages
File Size Distribution
Size (MB)CountPercentage
0.0213,4275.8%
0.0312,1425.3%
0.0412,0855.3%
0.0511,8505.2%
0.019,9294.3%
StatisticValue
Range0 - 68.83 MB
Mean0.565 MB
Median0.15 MB
Standard Deviation1.134 MB

Metadata Completeness Crisis

FieldMissingPresentCompleteness
Subject182,43047,48720.6%
Author78,269151,64866.0%
Title51,514178,40377.6%
Created Date3,260226,65798.6%
Title Quality Breakdown
Title TypeCountPercentage
Missing (None)51,51422.4%
Generic "Document"11,6995.1%
"untitled"2,0810.9%
Meaningful titles~165,00071.6%
Top Authors
AuthorCount
U.S. Government Printing Office11,838
Unknown3,477
Administrator1,630
U.S. Government Accountability Office1,390
Top Subjects
SubjectCount
Extracted Pages11,692
NIOSH HHE REPORT466
CMS Opinion Template353
SEC Financial Proposals Summary230

Processing Errors

Error TypeCountPercentage
Could not read Boolean object4649.5%
cryptography>=3.1 required for AES1516.1%
Stream ended unexpectedly99.7%
'NullObject' has no attribute 'get'55.4%
Other errors1819.4%

Temporal Coverage

Date FieldRangeIssues
Modified Date1979-12-31 to 2025-03-31(dates in 2023-2025 are incorrect/defaulted to)
Created DateVarious formats1,573 invalid "D:00000101000000Z"

Critical Assessment

[!NOTE] Generated by Claude Sonnet-4, unsolicited (as always)
Data Quality Issues
IssueSeverityImpact
Metadata PovertyCRITICAL79% missing subjects kills discoverability
Title DegradationHIGH28% generic/missing titles
Date InconsistenciesMEDIUMInvalid formats, future dates
Processing ErrorsLOW0.04% failure rate acceptable
Key Insights

Document Profile: Typical government PDF = 10 pages, 0.15 MB, metadata-poor

Fatal Flaw: This dataset has excellent technical extraction (99.96% success) but catastrophic intellectual organization. You're essentially working with 230K unlabeled documents.

Bottom Line: The structural data is solid, but without subject classification for 79% of documents, this is an unindexed digital landfill masquerading as an archive.