Offlin33er/qwen25-coder-7b-solidity-audit

Qwen2.5-Coder-7B — Solidity Audit Adapter (QLoRA, r=32)

A LoRA adapter for Qwen/Qwen2.5-Coder-7B-Instruct fine-tuned to produce structured Solidity security-audit reports: given a contract snippet, it emits ## Finding: <type> with functionality analysis, Description, and Recommendation — or a clear "no vulnerability identified" verdict.

Trained as part of the Solidity Audit Scanner project.

Newer versions available: V2 (canonical 8-class taxonomy, 48.5% type match) and V3 (verified fallback resolution, best detection F1 0.986). This V1 card documents the original run.

Training

Training loss (trackio curve): 1.21 @ step 5 → 0.87 @ step 15 → 0.64 @ step 48 → 0.56 @ step 115 → ~0.52 at the end of the cosine tail; token accuracy 0.71 → 0.86.

Evaluation (measured, n=100 stratified)

Stratified sample of the held-out test split — 70 vulnerable / 30 clean rows, seed 42, greedy decoding, exact training-time system prompt, 512 new tokens. Full per-row records in `eval_results.json`.

Read this honestly:

• —The perfect detection score is largely a dataset artifact. The held-out rows come from real audit reports where vulnerable snippets exhibit strong surface signals (the specific flawed construct the report was about), and clean rows are curated non-vulnerable code. Separating the two is much easier than real-world auditing, where the hard part is finding what nobody reported. Do not read F1 = 1.0 as "production-ready auditor."
• —The useful negative result is the 30% type match. The model detects that code is flawed reliably but labels the flaw coarser than the reference taxonomy (e.g., reference "integer overflow" → predicted "arithmetic error"; reference "reentrancy" → predicted "frontrunning"). Improving fine-grained typing needs a larger, normalized-type training set — the dataset's ~200 long-tail types have ≤6 examples each.
• —Two prior eval rounds (r2: mismatched system prompt + 256-token truncation → 53% unparseable; r3/r4: corrected protocol) are recorded in the repo history; r4v2 (greedy, deterministic) reproduced r3's counts and persisted per-row records with sample generations.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-Coder-7B-Instruct", dtype=torch.bfloat16, device_map="auto"
)
model = PeftModel.from_pretrained(base, "Offlin33er/qwen25-coder-7b-solidity-audit")
tok = AutoTokenizer.from_pretrained("Offlin33er/qwen25-coder-7b-solidity-audit")

messages = [
    {"role": "system", "content": "You are a senior smart-contract security auditor. Audit the provided Solidity code. If a vulnerability is present, report it as '## Finding: <type>' followed by the functionality, a Description, and a Recommendation. If none is found, state that no vulnerability was identified and describe what the code does."},
    {"role": "user", "content": "Audit the following Solidity code for security vulnerabilities.\n\n```solidity\n<YOUR CODE>\n```"},
]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(tok.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Limitations

• —Trained on ~2.5K audited snippets — narrow coverage; ~200 vulnerability types appear ≤6 times each.
• —Sequence length 2048 truncates some longer descriptions during training.
• —Eval was n=100 stratified from the held-out split, not a public benchmark; no cross-benchmark score is claimed.
• —It does not replace a professional audit. Findings are candidates for human review.

Defensive security tooling: analyzes code you paste in. Only audit contracts you are authorized to review.