govdocs
Datasets
All datasets matching “govdocs”govdocs1-pdf-source
govdocs1: source PDF files
[!NOTE]
Converted versions of other document types (word, txt, etc) are available in this repo
This is ~220,000 open-access PDF documents (about 6.6M pages) from the dataset govdocs1. It wants to be OCR'd.
Uploaded as tar file pieces of ~10 GiB each due to size/file count limits with an index.csv covering details
5,000 randomly sampled PDFs are available unarchived in sample/. Hugging Face supports previewing these in-browser, for example this one… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-pdf-source.govdocs1-by-extension
govdocs1 Dataset: By File Extension
[!NOTE]
PDFs from govdocs1 are at this repo in "raw" file form - no simple "mostly correct" way to convert to text
Markdown-parsed versions of documents in govdocs1 with light filtering.
Usage
Load specific file formats (e.g., .doc files) parsed to markdown with pandoc:
from datasets import load_dataset
# Replace "doc" with desired config name
dataset = load_dataset("BEE-spoke-data/govdocs1-by-extension", "doc")… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-by-extension.govdocs1-image
BEE-spoke-data/govdocs1-image
This contains .jpg files from govdocs1. Light deduplication was applied (i.e. jdupes on all files) which removed ~500 duplicate images.
DatasetDict({
train: Dataset({
features: ['image'],
num_rows: 108895
})
})
source
Source info/page: https://digitalcorpora.org/corpora/file-corpora/files/
@inproceedings{garfinkel2009bringing,
title={Bringing Science to Digital Forensics with Standardized Forensic Corpora}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-image.govdocs1-txt-raw
Dataset Card for "govdocs1-txt-raw"
Somewhere to put the raw txt files before filtering them
Source info/page: https://digitalcorpora.org/corpora/file-corpora/files/
@inproceedings{garfinkel2009bringing,
title={Bringing Science to Digital Forensics with Standardized Forensic Corpora},
author={Garfinkel, Simson and Farrell, Paul and Roussev, Vassil and Dinolt, George},
booktitle={Digital Forensic Research Workshop (DFRWS) 2009},
year={2009},
address={Montreal, Canada}… See the full description on the dataset page: https://huggingface.co/datasets/BEE-spoke-data/govdocs1-txt-raw.wpml-eval-govdocs
WPML-Eval: Govdocs HTML → WordProcessingML 변환 평가 데이터셋
Govdocs1 공개 말뭉치에서 추출한 1,000개 영문 HTML 문서와 대응하는
단순화된 WordProcessingML(Office Open XML 본문 포맷) 정답 파일로 구성된
연구/평가 전용 데이터셋입니다.
구성
버킷
문서 수
설명
1k-3k_tokens
100
입력 HTML ≈ 1,000–3,000 tokens
3k-4k_tokens
100
4k-5k_tokens
100
5k-6k_tokens
100
6k-7k_tokens
100
7k-8k_tokens
100
8k-9k_tokens
100
9k-10k_tokens
100
10k-11k_tokens
100
12k-13k_tokens100
합계
1,000
각 버킷당 고유 문서
각 버킷 폴더에는 세 가지 하위… See the full description on the dataset page: https://huggingface.co/datasets/timtkddn/wpml-eval-govdocs.