SASVAAI/Gemma4-12b-defect-detection

gemma-4-12B Defect Detection (LoRA)

Classifies the source of a single C function as defective or clean, for triaging C code for security defects.

This is a LoRA adapter for google/gemma-4-12B, trained with bf16 LoRA (no quantisation) via TRL SFT.

Model details

Trainable parameters: 65,568,768 across 328 modules — 0.548% of the base. The adapter file is 262,373,216 bytes (656 tensors: lora_A + lora_B per module). All 656 sit under model.language_model; the base model's vision and audio towers are loaded but untouched. exclude_modules is null in adapter_config.json and none was needed — unlike the 26B/31B Gemma 4 variants, this unified checkpoint has no vision_tower submodule for the seven names to match against.

Of the 48 text layers, 40 carry a `v_proj` adapter and 8 do not. This is not a gap in coverage: layers 5, 11, 17, 23, 29, 35, 41 and 47 are the full_attention layers (config.json → text_config.layer_types), and the base model sets attention_k_eq_v: true, so those layers ship no separate v_proj weight at all. Every v_proj that exists in the base — 40 of 40 — is adapted.

Intended use

Direct use. Binary classification of one C function at a time. The model was trained on a specific prompt shape and that shape is part of the contract:

• —google/gemma-4-12B ships no chat template (tokenizer.chat_template is null on both the base and this adapter's tokenizer). Training and evaluation therefore used a plain-text prompt, assembled by hand — see How to get started for the exact string. Do not call apply_chat_template; there is nothing for it to apply.
• —The instruction is fixed and verbatim: "Classify whether the following C function contains a security defect. Answer with exactly one word: defective or clean."
• —The function source is wrapped in a triple-backtick fence.
• —The label is the first line of the generation, lowercased; discard anything after it.

Out of scope.

• —Not a security gate. A clean verdict is weak evidence. Use the model to prioritise review, not to sign off on code.
• —Localising, explaining, or repairing a defect. It emits one word and no rationale, and was never trained to produce an explanation.
• —Any specific CWE or vulnerability class. The label is an undifferentiated binary inherited from the CodeXGLUE corpus.
• —Languages other than C, and multi-function or whole-file inputs. Every training and eval example is a single C function.
• —Not a general-purpose assistant. It emits a bare label, never prose, and will degrade on open-ended chat.

How to get started

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-4-12B"
ADAPTER = "SASVAAI/Gemma4-12b-defect-detection"

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="bfloat16", device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

func = """static void unix_process_msgfd(CharDriverState *chr, struct msghdr *msg)
{
    TCPCharDriver *s = chr->opaque;
    struct cmsghdr *cmsg;
    /* ... */
}"""

INSTRUCTION = (
    "Classify whether the following C function contains a security defect. "
    "Answer with exactly one word: defective or clean."
)
SYSTEM = (
    "You are a security auditor for C/C++ code. Given a function, determine "
    "whether it contains a security vulnerability (such as missing bounds "
    "checks, NULL pointer dereferences, buffer overflows, use-after-free, "
    "integer overflows, or other subtle defects). Respond with exactly one "
    "word: `defective` if the function contains a vulnerability, or `clean` "
    "if it does not."
)

# This model has NO chat template. The prompt is this literal string — the same
# one the training and eval harness built. Reproduce it exactly.
prompt = f"<|system|>\n{SYSTEM}\n<|user|>\n{INSTRUCTION}\n\n```\n{func}\n```\n<|assistant|>"

inputs = tokenizer(
    prompt, return_tensors="pt", truncation=True, max_length=4096,
).to(model.device)

out = model.generate(**inputs, max_new_tokens=16, do_sample=False)
text = tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(text.strip().splitlines()[0].strip().lower())
# -> clean

Decoding matters. This model was evaluated with greedy decoding (do_sample=False, max_new_tokens=16) inside a 4096-token input window. A prompt that overflows that window is not truncated at either end — both ends are load bearing, the instruction at the head and the <|assistant|> cue at the tail — so the harness elides the middle of the function source and substitutes the literal marker /* ... source elided ... */, keeping the surrounding scaffolding intact. One of the 246 holdout rows needed this. Sampling will not reproduce the reported numbers.

The base model is ~24 GB in bfloat16 and fits on one 80 GB-class GPU.

Training details

Data. CodeXGLUE CC Defect Detection (Devign — C functions from QEMU and FFmpeg), rendered to flat {instruction, input, output} JSONL: instruction is the fixed directive above, input is the raw func field, output is the binary target rendered as the word defective or clean.

The upstream train and validation splits were concatenated and re-split. The harness pooled them into 24,586 rows and drew a 1% random holdout (val_split_ratio = 0.01), giving 24,340 train / 246 validation.

The loss is computed over the whole sequence, including the C source — the model was trained to predict the function text as well as the label. That is part of why the prompt shape is load-bearing.

Method

bf16_lora trains and evaluates without quantisation, so the score below is a clean bf16 measurement with no quantise/dequantise mismatch between training and inference.

No refinement stage ran; the published weights are the SFT adapter.

Final hyperparameters

Effective batch size: 16 (1 x 4 x 4). Optimizer steps: 2,918.

Every remaining knob sat at its default and is omitted rather than printed: use_dora, use_rslora, use_liger_kernel and use_sample_packing were all false, lora_init was default, loraplus_lr_ratio 1.0 and neftune_noise_alpha 0.0. KD parameters are omitted deliberately — this is a bf16_lora run, not a KD run, so kd_alpha/kd_beta/kd_temperature carry inert defaults that would imply distillation that did not happen.

Observed training metrics.

Loss falls from 1.2440 at step 10 to 0.5481 by step 50, then improves slowly: 0.4544 at the half-epoch mark, 0.4099 at the end of epoch 1, and 0.3561 by step 2,910. Most of the task is learned in the first 2% of epoch 1; the second epoch buys roughly 0.05 of training loss.

Evaluation

Protocol. All 246 holdout rows, no sampling. Predictions generated greedily (do_sample=False, max_new_tokens=16) with the base loaded in bf16 and the adapter applied — the adapter is not merged before evaluation. The predicted label is the first line of the generation, lowercased and matched against the two valid labels; anything matching neither counts as unparsed and scores as wrong. F1-macro is computed over exactly the two labels present in the references, so absent classes cannot inflate it. Prompts were built inside a 4096-token input window with middle-elision on overflow (1 of the 246 rows), and generated in left-padded batches over length-sorted prompts.

This is a validation split, not a held-out test set. Model selection used this split, so expect some optimistic bias. It is also small (246 examples) — expect sampling noise on the order of +/-0.03 F1-macro. And because the upstream train and validation sets were pooled before the holdout was drawn, these numbers are not comparable to published CodeXGLUE results.

Limitations and bias

Domain narrowness. C functions from the Devign corpus (QEMU and FFmpeg). Expect a substantial drop on other codebases, other C styles, and any other language — none of which were measured.

Inherits all biases and limitations of the base model. This adapter changes 0.548% of the parameters and was not evaluated for social bias, safety, or fairness across demographic groups. It is a code classifier and was not assessed for adversarial inputs — a function crafted to look benign is untested.

Environmental impact

Covers this run only.

Framework versions

• —PEFT 0.18.1
• —TRL: 1.0.0
• —Transformers: 5.16.0.dev0
• —Pytorch: 2.5.1+cu121
• —Datasets: 4.8.4
• —Tokenizers: 0.23.1

transformers is a git-main build: gemma4_unified is not in the stable PyPI release.

Citation

@misc{gemma4_12b_defect_detection_2026,
  title  = {gemma-4-12B Defect Detection (LoRA)},
  author = {{SASVA AI Model Cognition Labs (MCL) Team}},
  year   = {2026},
  url    = {https://huggingface.co/SASVAAI/Gemma4-12b-defect-detection}
}