Alibaba-NLP/SecRespond

SecRespond

<p align="center"> 💻 <a href="https://github.com/Alibaba-NLP/qqr/tree/main/data/secrespond">GitHub</a>&nbsp; | &nbsp; 🤖 <a href="https://modelscope.cn/datasets/iic/SecRespond">ModelScope</a>&nbsp; | &nbsp; 📄 <a href="https://arxiv.org/abs/2607.26791">Paper</a> </p>

Introduction

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response evaluates whether an AI agent can investigate a compromised host after an attack has already succeeded.

For each cyber range, the responder receives a frozen forensic disk snapshot together with synthetic host-security-product outputs, then produces an evidence-backed incident analysis and remediation plan. The benchmark focuses on post-compromise investigation and response rather than pre-compromise vulnerability discovery.

Package Layout

Paths below are relative to the dataset repository root. Each range includes its metadata and one independently verifiable forensic disk archive:

.
├── LICENSE
├── README.md
├── task-prompts/
│   ├── linux.md
│   └── windows.md
├── evaluation/
│   ├── SKILL.md
│   └── prompt.md
└── ranges/
    └── <range>/
        ├── checklist.md
        ├── checklist.en.md
        ├── sas-mock/
        ├── disk.tar.gz
        └── disk.tar.gz.sha256

The GitHub tree contains the same lightweight benchmark assets but intentionally excludes disk.tar.gz, disk.tar.gz.sha256, and expanded disk/ trees. This full dataset adds one archive and checksum per range.

The dataset contains two response-agent prompts, two evaluator files, and 10 ranges. Every range has an authoritative checklist.md, an aligned English checklist.en.md, one disk.tar.gz, and one disk.tar.gz.sha256; the available sas-mock files vary by range.

Do not invent placeholder sas-mock files when a category is absent. Expanded disk/ trees are runtime artifacts and are not included in the dataset.

Task and Evaluation Assets

Response Agent

The task-prompts/ directory contains two complete, standalone response-agent prompts:

Select exactly one prompt according to the range platform. Do not concatenate the Linux and Windows prompts.

Each responder prompt uses the following placeholders:

The responder writes five files:

All five files are required. run_task.sh returns exit code 4 when the model run succeeds but any required output is missing. run_evaluation.sh checks the complete set before creating its output workspace and exits before the model call if any report is missing.

Evaluator

The evaluator consists of a prompt plus the benchmark scoring Skill:

evaluation/
├── SKILL.md
└── prompt.md

Render prompt.md with these placeholders:

The evaluator writes:

• —evaluation-report.md: per-checkpoint scores, rationale, quoted evidence, and summaries.
• —scores.json: structured per-checkpoint detection and plan scores with an overall summary.

Restoring a Range Disk

Each range already includes disk.tar.gz and disk.tar.gz.sha256. From the dataset repository root, verify a selected archive before extracting it:

RANGE=ssh-miner
RANGE_DIR="ranges/${RANGE}"

(cd "$RANGE_DIR" && sha256sum -c disk.tar.gz.sha256 && tar -xzf disk.tar.gz)

The archive must extract beneath a single top-level disk/ directory:

ranges/ssh-miner/disk/

Expose the extracted directory to the responder as a read-only forensic workspace. A single aggregate archive containing every range is not a substitute for the per-range disk.tar.gz release contract.

Runtime Integration Contract

The prompt and evaluation files are static assets and are not discovered implicitly. A compatible harness must:

1. 1.Select task-prompts/linux.md or task-prompts/windows.md from the range platform.
2. 2.Render DISK_PATH, SAS_PATH, and OUTPUT_PATH into the selected responder prompt.
3. 3.Run the responder on an analysis machine with the forensic disk exposed read-only.
4. 4.Render all four evaluator placeholders and load evaluation/SKILL.md.
5. 5.Evaluate the responder outputs against the selected range's checklist.md.

The standalone dataset payload does not include executable runner code. The qqr repository provides two OpenCode runner scripts:

• —`scripts/secrespond/run_task.sh`: run the response task for one range.
• —`scripts/secrespond/run_evaluation.sh`: evaluate the completed response reports for one range.

Running these scripts requires OpenCode. Follow the official installation and configuration documentation, then run both scripts from the qqr repository root:

git clone https://github.com/Alibaba-NLP/qqr.git
cd qqr

DATA_DIR=/path/to/SecRespond
TASK_OUTPUT=/path/to/task-output
EVALUATION_OUTPUT=/path/to/evaluation-output
MODEL=alibaba-cn/qwen3.7-max  # Replace with a model configured in OpenCode.

scripts/secrespond/run_task.sh ssh-miner \
  --data-dir "$DATA_DIR" \
  --disk-path "$DATA_DIR/ranges/ssh-miner/disk" \
  --output-dir "$TASK_OUTPUT" \
  --model "$MODEL"

scripts/secrespond/run_evaluation.sh ssh-miner \
  --data-dir "$DATA_DIR" \
  --reports-dir "$TASK_OUTPUT" \
  --output-dir "$EVALUATION_OUTPUT" \
  --model "$MODEL"

--data-dir must point to the dataset root containing task-prompts/, evaluation/, and ranges/. The task runner defaults to ranges/<range>/disk and ranges/<range>/sas-mock beneath that root; use --disk-path or --sas-path when either input lives elsewhere. It reads the selected disk directory in place and does not copy or symlink it, so expose a read-only mount or an isolated copy when the source must be protected.

run_task.sh renders one standalone platform prompt. run_evaluation.sh renders all four evaluator placeholders and references evaluation/SKILL.md by its concrete path. Both scripts use OpenCode's normal configuration discovery; configure providers, endpoints, and credentials in OpenCode itself. --model selects a configured model and may be omitted when OpenCode has a default. Both scripts support --prepare-only for validation without a model call.

Older integrations built around the former nested agents/ layout, {{SKILLS_PATH}}, or {{RESPONDER_OUTPUT_PATH}} must be updated before use. The evaluator accepts only CHECKLIST_PATH, DETECTION_REPORTS_PATH, EVALUATION_SKILL_PATH, and OUTPUT_PATH.

Range Inputs

Each range supplies three logical inputs:

Each range also includes ranges/<range>/checklist.en.md as an aligned English translation. The runner-facing authoritative contract remains checklist.md.

SecRespond intentionally mixes alert-visible and silent artifacts. Complete investigations require proactive disk analysis rather than simply restating alerts.

Ranges

The current package contains 10 cyber ranges spanning four entry-point types, 21 MITRE ATT&CK techniques, and five operating systems.

Evaluation

Each range's checklist.md is the authoritative evaluation contract; checklist.en.md is its aligned English translation. SecRespond organizes incident-response capability into five categories:

Detection and plan are scored independently:

Some checkpoints are detection-only and therefore have no plan score; others are plan-only and therefore have no detection score. N/A dimensions are excluded from their respective aggregates. Quality (Q) items may be reported separately from capability aggregation as specified by the individual checklist. Do not combine Detection and Plan into a single score.

Citation

If you use SecRespond in your research, please cite:

@misc{wang2026secrespondbenchmarkingaiagents,
  title={SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response},
  author={Lehan Wang and Boli Chen and Ruixue Ding and Pengjun Xie and Jinwei Huang and Zhendong Liu and Shuo Wang and Tao Lei and Xin Ouyang and Xiaomeng Li},
  year={2026},
  eprint={2607.26791},
  archivePrefix={arXiv},
  primaryClass={cs.CR},
  url={https://arxiv.org/abs/2607.26791},
}

Ethics Statement

Third-Party Model Evaluation

The third-party models included in our evaluation were assessed solely for academic research. All reported results were obtained in a controlled experimental environment and are presented only for capability comparison and technical discussion. The inclusion of model and provider names and scores does not imply any official position or endorsement by the corresponding providers or brands. Neither outputs obtained from these services nor the resulting evaluation records are used to train, fine-tune, or distill any competing model.

Security Research and Use Restrictions

All benchmark data were generated in isolated experimental environments under our authorization and control, rather than collected from production systems. Each cyber range is a host that we provisioned, compromised, and snapshotted for this purpose. The released forensic and security-product artifacts are synthetic and contain no data from customer, production, or real-world business systems. We sanitized every disk snapshot by replacing real credentials, keys, and personal data with placeholders, retaining only the forensic artifacts required for analysis. The attack techniques, CVEs, and exploitation methods represented in the scenarios are derived solely from public knowledge bases and vulnerability disclosures, such as MITRE ATT&CK and public vulnerability databases. The benchmark contains forensic evidence rather than runnable exploit chains; it includes no proprietary or internally developed advanced attack techniques and no incremental attack information beyond public sources, and it cannot be directly used to conduct attacks against real-world systems. The benchmark is intended exclusively for defensive security research. Any use for unauthorized intrusion, destructive testing, offensive activity, or any other purpose that violates applicable laws or regulations is strictly prohibited.

License

The dataset files in this repository are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0). See LICENSE for the full license text.