GaviZhou/qwen3-vl-8b-i2c-hallu-v6-32k

Qwen3-VL-8B I2C-Hallu V6 32K

This is a full-parameter supervised fine-tune of Qwen/Qwen3-VL-8B-Instruct for visual code-error auditing. Given a ground-truth reference image and candidate plotting code, the model is trained to identify discrepancies and return a strict JSON object containing an errors array.

Intended use

Use this model for research and development workflows that compare a rendered or target figure with candidate code. The desired response format is:

{
  "errors": [
    {
      "category": "error category",
      "description": "description of the discrepancy",
      "code_span": {
        "start_line": 12,
        "end_line": 14,
        "anchor": "relevant code fragment"
      }
    }
  ]
}

code_span is optional. Line numbers are 1-based and should refer to the candidate code supplied in the prompt.

Training

• —Base model: Qwen/Qwen3-VL-8B-Instruct
• —Fine-tuning method: full-parameter SFT with ms-swift (tuner_type=full)
• —Precision: BF16
• —Context length: 32,768 tokens
• —Packing: bin packing to 32,768 tokens
• —Training data: 7,014 examples
• —Validation data: 373 examples
• —Data split: grouped by ground-truth image; ground-truth-image and candidate-code overlap between train and validation splits are both zero
• —Epochs / optimizer steps: 2 / 198
• —Effective batch size: 8 (2 devices × batch size 1 × gradient accumulation 4)
• —Optimizer settings: AdamW, learning rate 1e-5, weight decay 0.01
• —Learning-rate schedule: cosine decay with 5% warmup
• —Loss: assistant-response loss (loss_scale=last_round)

The checkpoint published here is the best checkpoint from the first training run: checkpoint-198.

Validation results

Teacher-forced validation metrics at checkpoint-198:

The packed validation set contains 42 sequences. These are likelihood and token-level metrics, not direct measurements of error-detection quality or hallucination rate. For model selection or production use, additionally evaluate generated answers on held-out examples with JSON/schema validity, error-category precision/recall/F1, and code-anchor/line-alignment metrics.

Quick start

Install a recent Transformers release with Qwen3-VL support, then load the model as an image-text-to-text model:

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "GaviZhou/qwen3-vl-8b-i2c-hallu-v6-32k"

model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

candidate_code = """# Candidate plotting code; retain 1-based line references."""
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "/path/to/ground_truth.png"},
            {
                "type": "text",
                "text": (
                    "Compare the ground-truth image with the candidate code. "
                    "Return only the required JSON object.\n\n"
                    f"Candidate code:\n{candidate_code}"
                ),
            },
        ],
    }
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=1024)
generated_ids = [
    output_ids[len(input_ids):]
    for input_ids, output_ids in zip(inputs.input_ids, generated_ids)
]
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])

The upstream Qwen3-VL model card contains additional examples and deployment guidance.

Limitations and responsible use

• —This model can produce incorrect, incomplete, or invalid JSON output. Parse and validate its output before using it in an automated workflow.
• —The reported validation metrics are not a substitute for task-level generation evaluation or human review.
• —It is not a general-purpose code verifier, security analyzer, or guarantee that a figure faithfully represents code.
• —Users are responsible for ensuring that their input images, code, and any downstream use comply with applicable licenses, privacy requirements, and policies.

License

This checkpoint is derived from Qwen/Qwen3-VL-8B-Instruct, which is released under the Apache-2.0 license. Users must also ensure that their use of the fine-tuning data and downstream inputs is permitted.

Citation

If you use this model, please also cite the base model:

@misc{qwen3technicalreport,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2025},
  eprint={2505.09388},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://arxiv.org/abs/2505.09388}
}