BEE-spoke-data/govdocs1-by-extension

govdocs1 Dataset: By File Extension

[!NOTE] PDFs from govdocs1 are at this repo in "raw" file form - no simple "mostly correct" way to convert to text

Markdown-parsed versions of documents in govdocs1 with light filtering.

Usage

Load specific file formats (e.g., .doc files) parsed to markdown with pandoc:

from datasets import load_dataset

# Replace "doc" with desired config name
dataset = load_dataset("BEE-spoke-data/govdocs1-by-extension", "doc")

Configurations

This dataset includes multiple configurations, each corresponding to a different file extension:

• —doc
• —docx
• —logs
• —ppt
• —pptx
• —rtf
• —txt

Each configuration contains train, validation, and test splits.

Dataset Details

• —Download Size: Varies by configuration
• —Dataset Size: Varies by configuration
• —Splits: Train, Validation, Test
• —Features: Section, Filename, Text

counts

Here's a summary of the number of examples for each configuration in the dataset (thus far):

1. 1.DOC Configuration
2. 2.Train Examples: 38,094
3. 3.Validation Examples: 1,002
4. 4.Test Examples: 1,003

1. 1.DOCX Configuration
2. 2.Train Examples: 141
3. 3.Validation Examples: 8
4. 4.Test Examples: 8

1. 1.Logs Configuration
2. 2.Train Examples: 9,223
3. 3.Validation Examples: 243
4. 4.Test Examples: 243

1. 1.PPT Configuration
2. 2.Train Examples: 13,865
3. 3.Validation Examples: 365
4. 4.Test Examples: 365

1. 1.PPTX Configuration
2. 2.Train Examples: 963
3. 3.Validation Examples: 53
4. 4.Test Examples: 54

1. 1.RTF Configuration
2. 2.Train Examples: 942
3. 3.Validation Examples: 52
4. 4.Test Examples: 53

1. 1.TXT Configuration
2. 2.Train Examples: 41,393
3. 3.Validation Examples: 1,089
4. 4.Test Examples: 1,090

Citation

@inproceedings{garfinkel2009bringing,
  title={Bringing Science to Digital Forensics with Standardized Forensic Corpora},
  author={Garfinkel, Simson and others},
  booktitle={Digital Forensic Research Workshop (DFRWS) 2009},
  year={2009},
  address={Montreal, Canada},
  url={https://digitalcorpora.org/corpora/file-corpora/files/}
}

For more detailed information on each configuration, refer to the dataset documentation.