mahend72/ICSThreatQA

ICSThreatQA

Dataset Summary

ICSThreatQA accompanies:

Rani, R., Kumar, M., Epiphaniou, G., and Maple, C. (2025). ICSThreatQA: A Knowledge-Graph Enhanced Question Answering Model for Industrial Control System Threat Intelligence. Expert Systems with Applications.

This release packages the benchmark's question-answer pairs into a single, canonical, machine-readable table (CSV and Parquet) suitable for loading with the Hugging Face datasets library, while preserving the original per-category Excel files unchanged for provenance.

Dataset Description

ICSThreatQA is a human-in-the-loop benchmark of question-answer pairs for Industrial Control System (ICS) cybersecurity threat intelligence, built from a threat knowledge base describing ICS-relevant entities (malware, threat groups, attack patterns, techniques) and the relationships between them. The benchmark contains 616 QA pairs across four question categories, verified directly against the raw .xlsx/.csv files in the source repository (see Known Limitations for how this count was derived).

Question Categories

Verified counts, calculated directly from the source workbooks:

These are question categories, not machine-learning train/test/validation splits.

Dataset Structure

Each row is one QA pair. Fields:

Stable ID Generation

IDs are generated deterministically as ICSQA-<C>-<NNNN>, where <C> is a single-letter category code (F/C/I/O) and <NNNN> is a 1-based, zero-padded sequence number assigned in the row order the record appears in its original source workbook (top to bottom, after Excel's own trailing fully-blank rows are dropped — see Known Limitations). Regenerating the dataset from the unmodified source files with scripts/prepare_dataset.py reproduces the exact same IDs every time; no reordering, sampling, or filtering is applied before assigning them.

Example Record

A real record from the dataset (not fabricated):

{
  "id": "ICSQA-F-0001",
  "question_type": "factual",
  "question": "What malware is associated with the 'Industroyer' attack?",
  "answer": "The 'Industroyer' malware is associated with the 'Industroyer' attack.",
  "response_gpt4o": "The 'Industroyer' malware is associated with the 'Industroyer' attack.",
  "response_gpt4": "Industroyer",
  "source_file": "factual.xlsx",
  "source_row": 2
}

Dataset Creation

Per the source repository's README, the QA pairs were constructed from a threat knowledge base of ICS entities, attributes, and relationships, with human-in-the-loop review used to keep the questions diverse and contextually grounded rather than templated. The response_gpt4o and response_gpt4 fields were present as columns in the original benchmark workbooks alongside each question and its reference (answer) field; this release does not add, alter, or interpret their original purpose beyond preserving them as auxiliary columns.

Separately, the source repository also includes model_responses/ (outputs from a knowledge-based RAG model, a hybrid model, and a keyword-based baseline run against this same question set) and experiment_results/ (zero-shot and few-shot evaluation spreadsheets for those approaches plus GPT-4o/GPT-4). These are not part of the canonical QA table in data/; a copy is included separately under artifacts/ for reproducibility (see Optional Evaluation Artifacts).

Human-in-the-Loop Construction

From the source repository: the QA pairs were constructed from a threat knowledge base of ICS entities, attributes, and relationships, with human-in-the-loop review used to keep the questions diverse and contextually grounded rather than templated.

AI Assurance Relevance

Beyond its original use in the accompanying paper, this benchmark's structure — a fixed, human-validated question set spanning four question types, with multiple candidate QA approaches (knowledge-based RAG, hybrid, keyword-based, general-purpose LLM) evaluated against it under both zero-shot and few-shot conditions — makes it usable as a testbed for broader AI assurance research questions, including:

• —Answer reliability — whether an approach consistently produces correct answers within a question category.
• —Grounding — whether knowledge-graph-grounded approaches produce answers traceable to the underlying threat knowledge base, versus ungrounded approaches.
• —Retrieval quality — for RAG/hybrid approaches, whether retrieved context is relevant before being used to generate an answer.
• —Reasoning consistency — particularly for inferential questions, whether reasoning stays consistent across similarly structured questions.
• —Trustworthy RAG — using the contrastive/inferential/opinion splits to probe reasoning and grounding behavior distinct from simple factual recall.
• —Human oversight — the benchmark's human-in-the-loop construction is a documented part of how it was built; whether human review is also needed downstream is a separate, open question this benchmark can help study.
• —LLM evaluation in security-sensitive domains — using this benchmark to evaluate any LLM-based QA system on ICS-specific threat intelligence content.

Important: these are broader research interpretations of what this benchmark and its accompanying artifacts support studying. They are not a claim that the original 2025 paper experimentally evaluated all of these assurance properties. Consult the paper itself for what was actually measured and concluded.

Potential Future Assurance Extensions

The following are future research directions this benchmark could support and are not existing experiments or results in this release:

• —Hallucination evaluation — measuring how often an approach produces claims unsupported by the knowledge base.
• —Evidence attribution — testing whether generated answers can be traced back to specific knowledge-graph entities or retrieved passages.
• —Calibration — assessing whether an approach's confidence matches its actual correctness rate.
• —Retrieval-failure analysis — characterizing cases where RAG/hybrid retrieval returns irrelevant or missing context.
• —Adversarial queries — constructing edge-case questions to probe failure modes not covered by the existing pairs.
• —Robustness testing — evaluating stability of answers under paraphrased or reworded versions of the same question.
• —Knowledge poisoning — studying the effect of corrupted or manipulated knowledge-base entries on downstream answers.
• —Indirect prompt injection in RAG — testing whether retrieved context can be crafted to hijack a RAG pipeline's output.
• —Human–AI escalation/review policies — defining when a QA system's output should be routed to a human analyst, using this benchmark's question types as a starting taxonomy.

Reproducibility

Regenerate the canonical table from the raw source files:

pip install -r requirements.txt
python scripts/prepare_dataset.py
python scripts/validate_dataset.py

prepare_dataset.py reads the four raw workbooks in raw/, validates their structure, combines them, assigns provenance fields and stable IDs, writes data/icsthreatqa.csv and data/icsthreatqa.parquet, and prints a validation report. validate_dataset.py independently re-derives the dataset and checks it against the checked-in outputs (record counts, missing values, duplicates, category values, CSV/Parquet equality, and deterministic regeneration), failing clearly if anything doesn't match.

Loading With Hugging Face

After this dataset is published to the Hugging Face Hub, it should be loadable as:

from datasets import load_dataset

dataset = load_dataset("<HF_USERNAME>/ICSThreatQA")

Replace <HF_USERNAME> with the actual namespace once this dataset has been published — the value above is a placeholder.

Known Limitations

• —Total record count is 616. The four raw workbooks contain 316 (factual) + 100 (contrastive) + 100 (inferential) + 100 (opinion) = 616 records. This was verified directly against raw .xlsx row counts (after excluding a trailing block of fully blank, unformatted rows in factual.xlsx — an Excel formatting artifact, not data) and against the .csv mirrors already present in the source repository.
• —Duplicate questions exist in the original benchmark and are preserved. factual.xlsx contains 4 questions that each appear twice (2 of these are full duplicate rows with identical answers); contrastive.xlsx contains 9 questions that each appear 2–3 times. These are part of the original benchmark data and were not removed or deduplicated.
• —A few cells in `factual.xlsx` were auto-formatted by Excel as numbers or dates instead of text (e.g. a bare year like 2020, or an answer written as "December 1, 2016" that Excel reinterpreted as a datetime). These were normalized to plain text (ISO date or digit string) for consistent dtypes in the published table; the underlying values were not altered. See scripts/prepare_dataset.py (_to_text) for the exact rule.
• —The response_gpt4o and response_gpt4 fields are auxiliary responses captured in the original workbooks alongside each question; how or why they were generated is not documented in the source repository beyond what is stated above, and should not be assumed to represent a rigorous, separate evaluation run.
• —No dataset-level split (train/validation/test) is defined; question_type is a content category, not an ML split.

Responsible Use

This benchmark is intended for research and evaluation purposes. Answers in this dataset — whether the reference answer field or the auxiliary model responses — should not be used as a substitute for validated professional security analysis in an operational ICS environment. Any use of QA systems evaluated on this benchmark in a real security context should retain qualified human review before acting on their outputs.

Citation

If you use this dataset, please cite the original paper:

@article{rani2025icsthreatqa,
  title={ICSThreatQA: A Knowledge-Graph Enhanced Question Answering Model for Industrial Control System Threat Intelligence},
  author={Rani, Ruby and Kumar, Mahender and Epiphaniou, Gregory and Maple, Carsten},
  journal={Expert Systems with Applications},
  pages={130180},
  year={2025},
  publisher={Elsevier},
}

Still needs verification: volume, issue, and DOI are not present in the source repository's metadata (README.md, CITATION.cff) and are intentionally omitted above rather than invented. Verify these against the published version on the Expert Systems with Applications / Elsevier website before relying on this citation.

Source Repository

<https://github.com/mahend72/ICSThreatQA-Dataset>

Optional Evaluation Artifacts

artifacts/ contains unmodified copies of files from the source repository that are not part of the canonical QA table but support reproducing the original comparisons:

• —artifacts/model_outputs/ — raw outputs from the knowledge-based RAG, hybrid, and keyword-based approaches run against this question set (copied unchanged from model_responses/).
• —artifacts/evaluation_results/ — zero-shot and few-shot evaluation spreadsheets for all evaluated approaches, plus a combined evaluation summary (copied unchanged from experiment_results/).

These files are provided for reproducibility only; they were not used to build, and are not mixed into, data/icsthreatqa.csv / data/icsthreatqa.parquet.