bebrws/security-8b

k3-sec-8b (v8)

An 8B cybersecurity agent (offense + defense) fine-tuned from Qwen/Qwen3-8B on 2197 verified agentic + Q&A traces, distilled from GLM-5.2 (round 4) and Kimi K3/K2.6 (rounds 1–3). Trained to operate an autonomous security harness — plan, run bash, read observations, write files, finish — not just answer security questions.

Pipeline per iteration: failure analysis on eval traces → parameterized, decontaminated seed factory → teacher best-of-3 rejection-sampled agentic transcripts (judged + artifact-checked) → full-FT SFT → multi-run attack/defend eval → next round.

Version 8 highlights

• —Data: 2197 unique rows (2103 agentic + 94 Q&A). Round 8 was a 136-trace DNS top-up (75 decode + 75 detector-contract, $8). Round 7 was the broad-coverage round: 985 traces across ALL ten eval skills (flaky six weighted 100-120, solid four kept warm at 60-80, 80 generalization). Round 6 added 564 GLM-5.2 traces targeting the five v3 tasks that never passed, generated from 580 parameterized seeds with decontamination-by-construction (every eval-graded string is blacklisted and asserted absent). Best-of-3 rejection with a glm-4.7-flash judge (kept 97%). Note: v3's advertised 157 rows contained only 135 unique after legacy merge duplicates; v4 is a genuine 5.2× data increase.
• —Training: full FT bf16, 2 epochs, lr 1e-5 cosine, eff. batch 32, seq 8192, adamw8bit, 5.37M tokens, ~42 min on 1× A100-80GB. **trainloss 0.616 · token-acc 87.2%** (v7: 0.627 / 86.6%, v6: 0.838 / 83.1%, v3: 1.395 / 69.1%). Ships with Qwen3 YaRN rope_scaling for 131072-token serving.
• —Eval (fixed 10-task synthetic attack/defend lab, agentic harness, 3 runs): mean 8.0/10 with ZERO variance (8, 8, 8) -- attack side 5/5 in all three runs (atk-dns fixed by the top-up). Defend side: bruteforce/webshell(2/3)/harden solid, def-detect-dns 1/3, def-ioc regressed to 0/3 (round-9 target).
• —MMLU spot check (60 questions, temp 0, same harness): v8 0.533 vs v7 0.550 vs base Qwen3-8B 0.550 -- general capability statistically indistinguishable from base. NO capability collapse from the agentic diet.

Per-task pass rates (P across runs)

Run-to-run variance is significant at temperature 0.7; single-run scores are not meaningful for this suite. Known v4 gap: atk-sqli — the model prefers to start the staged vulnerable app and fuzz it over HTTP instead of reading the offline artifacts (trace-verified behavioral prior, targeted in round 5).

Usage

vLLM, short-task/eval serving (disable static YaRN):

python3 -m vllm.entrypoints.openai.api_server \
  --model bebrws/k3-sec-8b --revision v7cti \
  --port 8000 --hf-overrides '{"rope_scaling":null}' --max-model-len 32768

Long-context serving: omit --hf-overrides and set --max-model-len 131072. Recommended sampling for agentic loops (non-thinking): temperature=0.7 top_p=0.8 top_k=20 min_p=0, chat_template_kwargs.enable_thinking=false, per-step completion cap ~4096 tokens.

Tool / function calling

Supported. The chat template accepts a tools argument (OpenAI-style JSON function schemas) and renders them into the system turn inside <tools></tools>. The model emits calls as:

<tool_call>
{"name": "<function-name>", "arguments": {<args-json-object>}}
</tool_call>

Multiple calls may be emitted in a single assistant turn. Return each result as a message with role: "tool"; the template renders it as <tool_response>…</tool_response>, and consecutive tool messages are merged into one user turn.

messages = [{"role": "user", "content": "Scan 10.0.0.5 for open ports"}]
tools = [{
    "type": "function",
    "function": {
        "name": "exec_shell_command",
        "description": "Run a shell command and return its output",
        "parameters": {
            "type": "object",
            "properties": {"command": {"type": "string"}},
            "required": ["command"],
        },
    },
}]

text = tokenizer.apply_chat_template(
    messages, tools=tools, add_generation_prompt=True, tokenize=False
)

vLLM serving with native tool-call parsing:

python3 -m vllm.entrypoints.openai.api_server \
  --model bebrws/k3-sec-8b \
  --enable-auto-tool-choice --tool-call-parser hermes

llama.cpp requires --jinja for the embedded template (and therefore tool calls) to be used.

Intended use & limitations

Defensive/offensive security research artifact, evaluated on a small synthetic lab. Not for: real intrusion activity, exploit weaponization, unsupervised security decisions, or non-security tasks. Outputs require qualified human review. Attack-side competence is deliberately scoped to CTF/lab-grade tasks.

Version history (8B lineage)

Weights are Apache-2.0 per the Qwen3 base; training traces were generated by GLM-5.2 and Kimi teachers and filtered by automated judging.

My main question

Did k3-sec-8b iterations beat their base model?

Answer: Yes — by v6, clearly. But early iterations were worse than base.

The k3-sec-8b line starts training from `Qwen/Qwen3-8B` (per docs/training-history.md). All numbers below are on the project's fixed 10-task agentic lab (5 attack + 5 defend, identical sampling conditions).

Details

• —v6 vs base (head-to-head, 3 runs each): v6 wins 27/30 vs 24/30.
• —Attack: v6 is a perfect 15/15 (all 5 attack tasks, all 3 runs); base is 12/15 (atk-dns failed all 3 runs).
• —Defense: tied 12/15 both (def-detect-dns fails for both; base also drops def-webshell/def-ioc once each).
• —Consistency: v6 scores 9/10 on every run; the base declines 9 → 8 → 7 across runs.
• —The training took several iterations to pay off. v1 (4/10), v2 (6/10), and v3 (mean 4.0/10 across 4 runs) all scored below the base — early SFT rounds initially hurt the strong base model before later rounds (agentic file-writing data, failure-targeted rounds, GLM-5.2 bulk traces) pushed v6 above it.
• —Context: the Qwen3-8B base is itself unusually strong on this lab (24/30) — stronger than Foundation-Sec-8B-Instruct (8/30) and RedSage-Qwen3-8B-taught (16/30) measured on the same benchmark. Beating it at all is a meaningful bar.

Sources

• —data/eval_cmp_base_r{1,2,3}.json — Qwen3-8B base runs
• —data/eval_cmp_student_r{1,2,3}.json — k3-sec-8b v6 runs
• —docs/training-history.md — v1–v3 iteration evals (data/eval_8b*.json)

Aside

Also interesting: base Qwen3-8B is itself very strong on this lab (24/30 = 80%) — stronger than FSec-Instruct (8/30) and stronger than RedSage-taught (16/30)! That's a notable context point for the report: the k3-sec-8b v6 is the strongest model evaluated on this lab so far.

External comparison: k3-sec-8b v7cti vs Foundation-Sec-1.1-8B-Instruct (Q8_0 GGUF)

Comparison note: Foundation-Sec-1.1-8B-Instruct (Cisco Foundation AI, Aug 2025) appears to be the closest cutting-edge cybersecurity-specialized instruct model to compare against — same 8B class, instruction-tuned, security-domain. Both models were evaluated in their Q8_0 GGUF format (the most similar quantized format available for each), served via vLLM on identical RTX 4090 hardware with identical sampling. Full report: RunPod evaluation, 2026-08-02.

Results (3 runs × 10 tasks = 30 trials per model)

Per-task pass rates (passes / 3 runs)

k3-sec-8b wins or ties 8 of 10 tasks. Its standout is def-webshell (3/3 vs 0/3) — log analysis and firewall-rule writing requiring multi-step shell-tool operation. FSec-1.1's only decisive win is def-harden (3/3 vs 0/3) — single-shot SSH config editing where instruction-following suffices.

Key takeaways

• —k3-sec-8b is the more capable agentic model (nearly 2× the operational score), consistent with its training on agentic tool-use traces. It scores a stable 5/10 every run; FSec-1.1 is volatile (1–4/10).
• —FSec-1.1 has slightly stronger factual knowledge (perfect 45/45 vs 43/45 on the MCQ battery), consistent with its 5.1B-token cybersecurity CPT. But that knowledge doesn't translate to agentic capability on this harness.
• —Neither model solves atk-dns or def-ioc — the hardest tasks on this lab.
• —The Q8_0 GGUF format costs k3-sec-8b ~1 knowledge-quiz point vs bf16 (43 vs 44) but does not materially degrade agentic performance.