ethanolivertroy/nist-publications-raw

NIST Publications - Raw PDFs

596 NIST cybersecurity publications in original PDF format - Complete source data for the nist-cybersecurity-training dataset.

Dataset Description

This dataset contains the raw, unprocessed PDF files downloaded from the NIST Computer Security Resource Center (CSRC). These are the exact source documents used to create the NIST cybersecurity training dataset and fine-tune the HackIDLE-NIST-Coder model.

Contents

• —596 PDF documents (2.0 GB total)
• —metadata.json with titles, URLs, and download information
• —Covers documents published through October 2025

Document Series

Key Documents Included

• —SP 800-53 Rev. 5 - Security and Privacy Controls
• —SP 800-63-4 - Digital Identity Guidelines (July 2025 release)
• —NIST CSF 2.0 - Cybersecurity Framework (CSWP series)
• —SP 800-207 - Zero Trust Architecture
• —SP 800-37 Rev. 2 - Risk Management Framework
• —FIPS 140-3 - Cryptographic Module Validation
• —SP 800-161 Rev. 1 - Supply Chain Risk Management
• —SP 800-171 Rev. 3 - Protecting Controlled Unclassified Information

Data Pipeline

This raw dataset is part of a complete data processing pipeline:

1. Raw PDFs (this dataset)
   ↓
2. Extraction (Docling + MarkItDown)
   ↓
3. Training Data Preparation
   ↓
4. Training Dataset: ethanolivertroy/nist-cybersecurity-training
   ↓
5. Fine-tuned Models: HackIDLE-NIST-Coder v1.1

Usage

Download All PDFs

from datasets import load_dataset

# Download the dataset
dataset = load_dataset("ethanolivertroy/nist-publications-raw")

# Access metadata
with open("metadata.json", "r") as f:
    metadata = json.load(f)

# List all PDFs
pdf_files = [f for f in os.listdir(".") if f.endswith(".pdf")]
print(f"Downloaded {len(pdf_files)} PDFs")

Download Specific Document

from huggingface_hub import hf_hub_download

# Download a specific PDF
pdf_path = hf_hub_download(
    repo_id="ethanolivertroy/nist-publications-raw",
    filename="Digital Identity Guidelines_ Authentication and Authenticator Management.pdf",
    repo_type="dataset"
)

Reproduction

To reproduce this dataset (download fresh copies from NIST):

# Clone the source repository
git clone https://github.com/ethanolivertroy/nist-tuned-model

# Run the scraper
cd nist-tuned-model
python src/download_all_nist.py --output data/raw

The scraper downloads documents with these filters:

• —Series: FIPS, SP, IR, CSWP
• —Status: Final (published)
• —From: NIST Computer Security Resource Center

Metadata

The metadata.json file contains structured information for each PDF:

{
  "title": "Document title",
  "detail_url": "https://csrc.nist.gov/pubs/...",
  "local_path": "data/raw/filename.pdf",
  "downloaded": true
}

Related Resources

• —Training Dataset: ethanolivertroy/nist-cybersecurity-training - 530,912 processed examples
• —MLX Model: ethanolivertroy/HackIDLE-NIST-Coder-v1.1-MLX-4bit
• —GGUF Model: ethanolivertroy/HackIDLE-NIST-Coder-v1.1-GGUF
• —Ollama: etgohome/hackidle-nist-coder

License

CC0 1.0 Universal (Public Domain)

All NIST publications are in the public domain and not subject to copyright in the United States. This dataset is released under CC0 for maximum reusability.

Citation

@misc{nist-publications-raw,
  title={NIST Cybersecurity Publications - Raw PDFs},
  author={Troy, Ethan Oliver},
  year={2025},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/datasets/ethanolivertroy/nist-publications-raw}}
}

Acknowledgments

• —NIST Computer Security Resource Center - Original source of all documents
• —NIST Cybersecurity Framework Team - Framework development
• —NIST Privacy Framework Team - Privacy guidance

Dataset Statistics

• —Total Size: 2.0 GB
• —Document Count: 596
• —Date Range: 1977-2025 (primarily 2010-2025)
• —Format: PDF (various versions)
• —Languages: English

Version History

• —v1.1 (October 2025): Added CSWP series (28 documents), SP 800-63-4
• —v1.0 (Initial release): 568 documents (FIPS, SP 800/1800, IR)

Maintainer: Ethan Oliver Troy Contact: Via GitHub or HuggingFace profile Last Updated: October 2025