sajan-sarker/Qwen3.5-4B-LoRA-GRPO-CyberSec-Reasoner

🔐 CyberSec-Reasoner (Qwen3.5-4B | QLoRA + GRPO Fine-tuned)

A Three-Stage Post-Trained Reasoning LLM (Supervised Reasoning + GRPO Fine-Tuned) for Cybersecurity Vulnerability Analysis, optimized for CVE → CWE mapping with structured outputs.

A three-stage Supervised Instruction-Reasoning + GRPO fine-tuned LLM for cybersecurity reasoning, vulnerability classification, and explainable threat analysis.

Model Details

Model Description

CyberSec-Reasoner is a domain-specialized LLM designed to analyze cybersecurity scenarios using structured Chain-of-Thought reasoning.

• —🔍 CVE → CWE vulnerability mapping
• —🧩 Attack scenario decomposition
• —🛡️ Security reasoning with \<think>...\</think> traces
• —📚 Explicit Chain-of-Thought (CoT) reasoning for interpretable security analysis

Additional Information

• —Developed by: Sajan Kumer Sarker
• —Model type: Fully fine-tuned causal language model (merged from QLoRA + GRPO)
• —Language(s) (NLP): English
• —License: Apache License 2.0 (inherited from base model Qwen/Qwen3.5-4B by Alibaba)
• —Finetuned from model: Qwen/Qwen3.5-4B by Alibaba
• —Modifications: This model is a fine-tuned and merged version using QLoRA and GRPO for cybersecurity reasoning tasks.

Uses

Direct Use

This model is intended for:

• —Cybersecurity vulnerability analysis
• —CVE → CWE classification
• —Security reasoning and explanation generation
• —Educational use in cybersecurity and AI reasoning

Out-of-Scope Use

• —Fully automated security decision-making without human validation
• —Real-time production defense systems without human validation
• —High-stakes environments requiring guaranteed correctness
• —Malicious use such as generating exploit strategies

Bias, Risks, and Limitations

• —Limited coverage of CVE–CWE mappings
• —May produce incorrect or hallucinated CWE IDs
• —Performance relies more on reasoning than memorization
• —Dataset bias may influence predictions
• —Not all security domains are equally represented

Recommendations

• —Always verify outputs with authoritative security sources
• —Use alongside retrieval systems (RAG) for better accuracy
• —Do not rely on the model for critical security decisions
• —Combine with expert human analysis

How to Get Started with the Model

⚙️ This repository provides a fully merged model. No additional adapter loading is required. Use the code below to get started with the model.

Prerequisites

pip install --upgrade transformers==5.5.4

Option 1 — Quick Start with Wrapper Class (Recommended)

# Download the inference wrapper from the HuggingFace repo
import subprocess
subprocess.run(["wget", "https://huggingface.co/sajan-sarker/Qwen3.5-4B-LoRA-GRPO-CyberSec-Reasoner/resolve/main/inference.py"])

from inference import CyberSecReasoner, CyberSecReasonerConfig

cfg = CyberSecReasonerConfig()
model = CyberSecReasoner(cfg)

cve_description = """Analyze the following CVE description and map it to the appropriate CWE.
Provide a brief justification for your choice. Ensure the last line of your response contains only the CWE ID.
CVE Description: Cross-site scripting (XSS) vulnerability in MyBulletinBoard (MyBB) allows remote
attackers to inject arbitrary web script or HTML via a signature containing a JavaScript URI in the
SRC attribute of an IMG element..."""

messages = [{"role": "user", "content": cve_description.strip()}]

completion = model.generate(messages)
reasoning, final_output = model.format_completions(completion)

print("=== Reasoning Trace ===")
print(reasoning)

print("\n=== Analysis & CWE Classification ===")
print(final_output)

Option 2 — Direct Transformers Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "sajan-sarker/Qwen3.5-4B-LoRA-GRPO-CyberSec-Reasoner"

tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

cve_description = """Analyze the following CVE description and map it to the appropriate CWE.
Provide a brief justification for your choice. Ensure the last line of your response contains only the CWE ID.
CVE Description: <YOUR CVE DESCRIPTION HERE>"""

messages = [{"role": "user", "content": cve_description.strip()}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=1024, do_sample=False)

generated_tokens = outputs[0][inputs["input_ids"].shape[1]:]
print(tokenizer.decode(generated_tokens, skip_special_tokens=True))

Expected Response with option 1

=== Reasoning ===
<think>
Okay, let's tackle this CVE analysis. The problem is a cross-site scripting (XSS) vulnerability in MyBulletinBoard (MyBB). The description mentions that remote attackers can inject arbitrary web script or HTML through a signature with a JavaScript URI in the SRC attribute of an IMG element. The specific issue is using SGML numeric character references without a trailing semicolon.

First, I need to recall what XSS vulnerabilities ...
</think>

=== Analysis & CWE Classification ===
The CVE describes an XSS vulnerability caused by improper neutralization of user-controllable input (JavaScript URI in an IMG SRC attribute) embedded into HTML output. This directly aligns with ...

CWE-79

Training Details

Three-Stage Post-Training Pipeline

Training Procedure

Stage 1 — Knowledge-Based SFT

• —Goal: Teach foundational cybersecurity vocabulary and domain facts
• —Data: Merged dataset from Fenrir v2.0 + Cybersec-Reasoning-Merged
• —Method: SFT with full loss on prompt + assistant tokens (~96K samples, 3 epochs)

Stage 2 — Cold-Start Reasoning SFT

• —Goal: Establish the structured <think>...</think> reasoning format
• —Data: ~1,000 samples from Primus-Reasoning, filtered for CVE-to-CWE mapping
• —Method: SFT with assistant-only loss to enforce the reasoning structure

Stage 3 — GRPO Reinforcement Learning

• —Goal: Refine judgment and improve CWE prediction accuracy
• —Data: ~2,300 CVE-to-CWE samples from the Primus dataset
• —Method: GRPO with a composite reward function (see below)

Evaluation

Testing Data, Factors & Metrics

Testing Data

• —Held-out SFT dataset split
• —Dedicated CVE–CWE evaluation dataset with 1000 samples (cve-and-cwe-dataset-1999-2025)

Factors

• —Vulnerability type diversity
• —Reasoning structure correctness
• —Domain-specific keyword coverage

Metrics

• —BLEU
• —ROUGE
• —Perplexity
• —CWE Accuracy (exact match) [Custom rule based metric]
• —Format Match (\<think> ... \</think> structure) [Custom rule based metric]
• —Keyword Match [Custom rule based metric]

Results

• —BLEU: 0.1098
• —ROUGE1: 0.3680
• —ROUGE2: 0.1528
• —ROUGE3: 0.1779
• —ROUGELsum: 0.2507
• —PERPLEXITY: 1.7989
• —CWE Accuracy: 33.90%
• —Format Match: 94.85%
• —Keyword Match: 97.85%

*Note: \ Due to limited coverage of CVE–CWE knowledge and the relatively small 4B parameter size, the model may achieve lower exact-match accuracy for CWE classification. However, it is specifically optimized for structured reasoning on CVE–CWE tasks. As a result, when combined with external knowledge sources (e.g., a CVE–CWE database or RAG pipeline), the model is expected to produce more accurate mappings along with high-quality, interpretable reasoning.*

Summary

The model demonstrates strong reasoning capabilities but is limited by incomplete vulnerability knowledge coverage.

Model Examination

• —Structured reasoning trace analysis via \<think> blocks
• —Output post-processing for CWE extraction
• —Behavior shaped via reward-based RL (GRPO)

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• —Hardware Type: AMD MI300X (192GB single GPU)
• —Hours used: ~90 Hours
• —Cloud Provider: AMD Developer Cloud
• —Compute Region: Atlanta [Datacenter 1 - ATL1]
• —Carbon Emitted: Not estimated

Hardware Requirements

• —GPU recommended (>=16GB VRAM)
• —CPU inference possible but slow
• —Model size: ~9 GB

Model Card Authors

Sajan Kumer Sarker

Model Card Contact

For questions or collaboration, open an issue on the Hugging Face repository.

Framework versions

• —transformers>=5.5.4