ezesecops/rmf-ato-core

RMF/ATO Core Corpus

What this is

A curated corpus of current-revision NIST Risk Management Framework and authorization publications, plus a second tier of AI governance documents. Every row is source text — one control, one assessment objective, one RMF task, one AI RMF subcategory, one SSDF practice, one document section — carrying the identifiers a practitioner actually cites. It is built for retrieval and fine-tuning around authorization workflows: control selection, SSP and assessment work, categorization, continuous monitoring, and the emerging AI governance overlay.

5,510 rows, 1.9 MB, single `train` split. No embeddings, no synthetic Q&A, no system prompts.

Why another NIST dataset

Existing NIST corpora are typically indiscriminate scrapes, and they share four failure modes. Each is prevented here by construction rather than by cleanup:

The dataset is deliberately small. It is meant to be right, not exhaustive.

What's included

SP 800-53A content is extracted from the SP 800-53 Rev 5 OSCAL catalog's embedded assessment parts, not from a separate 53A file — NIST publishes no standalone 53A OSCAL artifact. Those rows are attributed to SP-800-53Ar5 and cite the catalog's hash.

Excluded

Superseded or withdrawn revisions (including 182 withdrawn SP 800-53 controls, each logged by name); NIST annual reports and workshop proceedings; pre-2010 legacy publications except FIPS 199/200, which remain in force; and draft publications. Also excluded from v1: CNSSI 1253 and DoD Instruction 8510.01, whose publishers block automated retrieval; and SP 800-60 Volume 2 Appendix E, which reproduces OMB memoranda and legislative provisions as wide reference tables — source material for the impact determinations rather than guidance, and it extracts as citation soup.

Schema

One row = one chunk.

Rows by chunk type

The 297 definition rows come from five glossaries: SP 800-37r2 (178), SP 800-137 (101), FIPS 199 (13), SP 800-18r2 (3) and FIPS 200 (2). The two large ones mark their entries typographically rather than with punctuation — SP 800-37 sets terms in bold, SP 800-137 marks each entry with a smaller bracketed source line — so both are read from the font, not from a TERM: pattern.

Example rows

{
  "id": "SP-800-53r5/control_enhancement/ac-2.3",
  "text": "AC-2(3) Account Management | Disable Accounts\nFamily: Access Control (AC) > AC-2 Account Management\n\nDisable accounts within [Assignment: organization-defined time period] when the accounts:\n(a) Have expired;\n(b) Are no longer associated with a user or individual;\n(c) Are in violation of organizational policy; or\n(d) Have been inactive for [Assignment: organization-defined time period].",
  "doc_id": "SP-800-53r5",
  "revision": "Rev 5 (OSCAL content version 5.2.0)",
  "tier": 1,
  "chunk_type": "control_enhancement",
  "control_id": "ac-2.3",
  "section_path": "AC > AC-2 > AC-2(3)"
}

{
  "id": "AI-100-1/ai_rmf_subcategory/govern-1.1",
  "text": "GOVERN 1.1: Legal and regulatory requirements involving AI are understood, managed, and documented.",
  "doc_id": "AI-100-1",
  "revision": "1.0",
  "tier": 2,
  "chunk_type": "ai_rmf_subcategory",
  "control_id": null,
  "section_path": "AI RMF Core > Govern > GOVERN 1.1"
}

Note the ODP rendering: {{ insert: param, ac-02_odp.01 }} in the OSCAL source becomes [Assignment: organization-defined …] / [Selection; one or more: …], the convention SP 800-53 itself prints. No template marker survives into any row.

How it was built

01 verify → 02 fetch → 03 parse OSCAL → 04 parse PDF → 05 chunk → 06 validate → 07 export

Each stage is an independently runnable, idempotent script. Source, tests, and the full rejection log live in the GitHub repository: <https://github.com/ezesecops/rmf-ato-core>

1,131 rows were rejected across the pipeline, every one recorded with a rule and a reason in rejections.jsonl. Rejected content is logged, never silently dropped.

Three review passes shaped this log after the pipeline first ran end to end. The first added the supply-chain rules that reject reference entries, running-header remnants and mid-sentence fragments. The second showed those rules were discarding rows that had real guidance underneath a damaged first line, so the pipeline now repairs what it can — a running-header remnant or a leading citation tag over substantive text is stripped and the row is published, and rows are only rejected when nothing substantive remains. The third recovered content the rules had been hiding: SP 800-137's glossary became 101 definition rows, and decorative drop caps stopped being read as headings. running_header_fragment rejected 101 rows before that repair step existed and rejects none now.

Provenance & integrity

Every artifact was retrieved once, hashed, and recorded. provenance.json ships with the dataset.

SP-800-53Ar5 rows cite the SP-800-53r5 catalog hash, because that is the artifact they were extracted from.

Limitations

• —PDF section coverage is partial and best-effort. Layout is not structure. Per-unit rows (controls, tasks, subcategories, practices, definitions) are high-confidence; section rows are the residue of heading detection. Content loss to furniture stripping is under 1% for most documents and about 8% for SP 800-218, whose bold bullet lists and two-line headings fragment worst.
• —Assessment objectives and methods are one row per control, not per leaf clause. A single determination statement ("account managers are assigned;") is not retrievable on its own. This keeps the corpus at ~5.5k coherent rows rather than ~13k fragments.
• —About 97 mid-sentence fragments were rejected rather than published. PDF page breaks, footnote interleaving and multi-column layout sometimes hand the extractor a passage that starts partway through a sentence (transparent the risk perceptions that organizations routinely use…). Those rows are dropped under midsentence_fragment and logged with the text that was discarded. Two consequences worth knowing: a small amount of real guidance — mostly in SP 800-30r1 and SP 800-37r2 — is missing from the corpus, and section coverage of those documents is therefore not continuous. Repairing the fragments would mean stitching text across page boundaries, which risks joining passages that were never adjacent; dropping them was the more conservative choice.
• —CNSSI 1253 and DoDI 8510.01 are absent from v1 — cnss.gov and esd.whs.mil block scripted retrieval, and their control tables were out of scope for v1 regardless.
• —SP 800-60 Volume 2 Appendix E is absent (see exclusions). The information-type entries it supports — 113 D.x rows — are present.
• —The AI RMF Playbook is a rolling web resource. Its rows reflect the version retrieved on the date above and will drift as NIST updates it.
• —Nineteen rows fell outside the length bounds and were rejected: one AI RMF subcategory (MAP 1.5, which the Playbook carries in full), two FIPS 199 glossary terms, and sixteen one-line glossary cross-references of the form "See authorization boundary."
• —This corpus reflects publications as of the build date. NIST revises documents, sometimes without notice — SP 800-18 Rev 1 was withdrawn six weeks before this build. Re-run the pipeline rather than assuming currency.
• —Not legal or compliance advice. These are reference texts; authorization decisions belong to the authorizing official.

License

Source documents are works of the United States Government and are in the public domain under 17 U.S.C. § 105. No copyright is claimed in them. The compilation, curation, manifest, and derived structure are released under CC0 1.0 Universal.

Maintainer & citation

Maintained by [@ezesecops](https://github.com/ezesecops) — <https://ezesecops.com>

@misc{rmf_ato_core_2026,
  author       = {Anene, Ebubeze},
  title        = {RMF/ATO Core Corpus: a curated, provenance-tracked NIST RMF and AI governance dataset},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/ezesecops/rmf-ato-core}},
  note         = {Built 2026-08-08 from current-revision NIST publications}
}

Found a bad row? That is the most useful thing you can report. Open an issue at <https://github.com/ezesecops/rmf-ato-core/issues> with the row id — every row traces back through sha256_source to the exact artifact it came from, so problems are reproducible.