aacudad/AnomalyThink-LLaVA-OneVision-7B-KCR

AnomalyThink-LLaVA-OneVision-7B-KCR

The strongest LLaVA-OneVision model in this family. Trained with SFT only, on a self-distilled corpus that the model family generated itself.

A LLaVA-OneVision-7B-SI model fine-tuned for explainable industrial anomaly detection (IAD). Given one product image it writes a reasoning trace in <think>, then a defect <location> and <type> when it sees a defect, and finally a binary <answer>. Research artefact from the MSc thesis Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection (TU Delft, 2026).

This is the cross-architecture replication of the thesis pipeline. The backbone is the same one IAD-R1 uses, so the comparison is on IAD-R1's own ground.

Results

Metric is balanced accuracy, (TPR + TNR) / 2, in percent.

Which build these weights are. This checkpoint is the first KCR build on this backbone. Its corpus was drawn from the whole 10,236-image rollout pool rather than from the 6,000-image SFT split (only 3,516 of its images lie inside that split, the other 2,484 come from the GRPO split) and it was balanced on the folder name rather than on the trace verdict, which left it at 45.0 % anomalous. The thesis reports the corrected build, rebuilt with all 6,000 images inside the SFT split and exactly 3,000 / 3,000 by verdict, as the replication result: 84.35 / 71.36, 87.32 / 72.65, 86.96 / 73.57 and 86.60 / 74.29 at epochs 1 to 4, with 87.32 / 72.65 the DS-MVTec-selected row. The first build stays on record in the thesis (Table 6.16, Appendix M). Both corpora are in the dataset repo: llava_kcr/sft_llava_C_train.json (first build, this model) and llava_kcr/sft_llava_C_corrected_train.json (corrected build).

Reference rows measured on the exact same harness:

Evaluation protocol. One shared harness for every row above. The DS-MVTec and VisA subsets of MMAD, single image per prompt, the same instruction the model was trained on, greedy decoding at temperature 0, at most 1024 new tokens, images capped at 262,144 pixels. Answers are parsed from the <answer> tag. Generation ran through vLLM 0.10.2, which agreed with the plain transformers generate path on 99 percent of a probe set. Nothing here is a re-scored or best-of-N number.

Strict scoring: a generation with no parsable verdict counts as wrong, as in the thesis. Six of the 2,141 VisA generations of the SFT then GRPO model have no verdict, which gives 72.38; over the 2,135 parsable outputs it is 72.58. No ordering changes.

Contamination note, please read this before you compare DS-MVTec numbers

The public LLaVA-OneVision training mixture (lmms-lab/LLaVA-OneVision-Data, config vision_flan(filtered)) contains 426 rows whose id matches %MVTecAD%. The base model has therefore seen MVTec-AD material during its own instruction tuning. Every DS-MVTec number for any LLaVA-OneVision derived model carries that caveat, including the 88.45 above, and including the IAD-R1 row. We do not know how much of the gap is real capability and how much is recall.

VisA is not affected. The same query over the mixture returns 0 rows for VisA. So the 74.25 on VisA is the clean number and it is the one to trust for a cross-model comparison.

Training

• —Base model: `llava-hf/llava-onevision-qwen2-7b-si-hf`.
• —Corpus: `llava_kcr/sft_llava_C_train.json`, 6,000 traces, balanced 50/50 over normal and anomalous parts, on Real-IAD images.
• —Recipe: supervised fine-tuning only. No reinforcement learning stage. The SigLIP vision tower is frozen, the multimodal projector and the language model are trained. Learning rate 1e-5, cosine schedule, warmup ratio 0.03, weight decay 0.1, effective batch size 32, context cutoff 8,192 tokens, bf16, 4 epochs.
• —Epoch: this is epoch 4 of 4 (step 748), the best of the four saved epochs. The full epoch curve on DS-MVTec / VisA was 85.29 / 72.32, 85.60 / 70.10, 86.45 / 73.56, and 88.45 / 74.25.

The KCR corpus here is LLaVA native, not borrowed from Qwen

This matters, so it is stated plainly. KCR stands for keep, correct, rewrite. The corpus for this model was built from LLaVA-OneVision's own rollouts, not from the Qwen rollouts used elsewhere in the thesis. The loop was:

1. 1.Sample 10,236 rollouts on Real-IAD training images from the sibling SFT then GRPO checkpoint, k = 8 per image at temperature 0.7.
2. 2.Bucket each trace by whether its verdict matched the label, into keep, needs correction, or needs rewrite.
3. 3.Score every trace with a Gemini-3-Flash faithfulness judge, and demote traces that were right for the wrong reason.
4. 4.Have Gemini-3-Flash correct or rewrite the traces that failed, then sample a balanced 6,000.

The intermediate arms are published next to the final one: `llava_kcr/` holds sft_llava_A_kept.json (keep only), sft_llava_B_kept_corrected.json (keep plus corrected), sft_llava_C_train.json (the first-build corpus this model was trained on) and sft_llava_C_corrected_train.json (the corrected build the thesis reports, see the build note above).

Because the rollouts come from this backbone, the corpus is on-policy for LLaVA-OneVision, and that is worth the extra loop. We also ran the control, which is the Qwen derived KCR corpus fine-tuned on this same LLaVA backbone with the same recipe. Its four epochs scored 87.70 / 70.97, 85.05 / 73.52, 86.97 / 69.95, and 85.85 / 70.75. No single checkpoint of that control reaches the 88.45 / 74.25 of the native corpus, and no checkpoint is best on both benchmarks at once. Building the corpus from the backbone's own rollouts is the difference.

Tested transformers versions

The weights and configs in this repo were written by transformers 4.51.3. Loading, the processor, and the full evaluation were verified under transformers 4.57.1, both through the plain HF generate path and through vLLM 0.10.2.

One warning for anyone rebuilding this pipeline. Checkpoints saved by transformers 5.0 write the rope settings under text_config.rope_parameters. Transformers 4.x does not read that key and silently falls back to rope_theta = 10000, which is 100 times too small. The model then stays fluent but goes blind and answers "no" to nearly everything, which looks like a collapsed run rather than a loading bug. This repo ships a plain text_config.rope_theta = 1000000.0 next to the 5.0 style block, so it loads correctly on both major versions.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration

repo = "aacudad/AnomalyThink-LLaVA-OneVision-7B-KCR"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

image = Image.open("part.png").convert("RGB")
product = "tile"          # the product category, it goes into the prompt

question = (
    f"Analyze the provided image of the {product}. "
    "Determine if there are any anomalies present. "
    "If an anomaly is detected, specify its type and location, "
    "and provide a detailed reasoning for your conclusion."
)
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": question},
]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(
    model.device, torch.bfloat16)

out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:],
                       skip_special_tokens=True))

Use this exact instruction. The model was trained on it and it degrades on a different phrasing.

Expected output on a defective part:

<think>
I am inspecting a tile with a speckled, grayish-white surface. ... In the center of the
tile, I detect a triangular, translucent plastic fragment. ...
</think>
<location>center</location>
<type>Contamination</type>
<answer>Yes</answer>

On a normal part the model emits <think> and then <answer>No</answer>, with no <location> or <type> tag.

Intended use and limitations

Research on explainable industrial anomaly detection. This is a thesis artefact, not a production inspection system.

Known limitations:

• —The DS-MVTec contamination caveat above.
• —The model can write a confident and well argued trace for a defect that is not there.
• —The <type> label is coarse and the model over-uses "missing parts" for any loss of material, including chips and gouges.
• —It was trained on Real-IAD style single-object images on plain backgrounds. Cluttered scenes, multiple parts per image, and very different lighting are out of distribution.
• —Reasoning traces were distilled from a teacher model. A fluent trace is not proof that the model looked at the right pixels.

Citation

@mastersthesis{acudad2026reasoning,
  author  = {Acudad, A.},
  title   = {Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection},
  school  = {Delft University of Technology},
  year    = {2026},
  type    = {Master's thesis},
  url     = {https://resolver.tudelft.nl/uuid:65c62420-79c0-447f-b095-7fb11d4474fc}
}

Thesis: <https://resolver.tudelft.nl/uuid:65c62420-79c0-447f-b095-7fb11d4474fc>. Code and evaluation files: <https://github.com/aacudad/IAD-VLMs>. The training data is at aacudad/AnomalyThink.

Related models

• —AnomalyThink-LLaVA-OneVision-7B-SFT
• —AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO
• —AnomalyThink-Qwen2.5-VL-7B-KCR, the same pipeline on the Qwen2.5-VL backbone

License

Apache-2.0, inherited from the LLaVA-OneVision-7B-SI base. Trained on Real-IAD images, which are not redistributed here, so cite Real-IAD separately. Reasoning traces were distilled from Gemini 2.5-Flash.