Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-lora

Qwen3-VL-4B Wiki-Screenshot QA LoRA — variable-k (1–6 images) @ 3x compression

LoRA adapter for Qwen/Qwen3-VL-4B-Instruct, fine-tuned as the reader in a retrieval-augmented Wikipedia-screenshot QA pipeline.

Unlike the earlier fixed-3-image adapters, this v2 reads any number of screenshots from 1 to 6 at inference time, at a fixed 3x pixel compression per image.

Performance (GPT-4.1 LLM-judge on 500 test examples)

Base@0x degrades sharply as the number of images grows (distractor confusion: 0.958 → 0.856 from k=1 to k=4). The 3x adapter is nearly k-invariant: 0.904 → 0.884 from k=1 to k=4 — only 0.020 drop.

At 3x compression, this adapter exceeds the uncompressed no-SFT baseline at k=2/3/4 (0.918/0.932/0.884 vs 0.912/0.892/0.856) and trails it only at k=1. The k=3 score (0.932) also beats the prior fixed-k=3 specialist ceiling (0.900) by +0.032 — a heavier 2× top3 oversample during training pushed multi-k flexibility past the single-k specialist.

Training setup

• —Method: LoRA (r=256, alpha=256, targets=all linear layers including ViT; freeze_vision_tower=false)
• —Base model: Qwen/Qwen3-VL-4B-Instruct
• —Framework: LLaMA-Factory (fork)
• —Optimizer: cosine LR, peak 1e-5, warmup 3%
• —Effective batch: 32 (perdevice=1 × 8 GPUs × gradaccum=4)
• —cutoff_len: 8192 (scaled per compression — larger for smaller compression where each image takes more tokens)
• —Hardware: 8× H100 80GB, DeepSpeed ZeRO-2, bf16
• —Training data: mixed 2× top3 (fixed k=3, 208k oversampled) + vark (k=1..6, 104k) at 2:1 ratio, 2 epochs over 312k samples per epoch. The released adapter is the best intermediate checkpoint at step 16000 (~1.64 epochs / 0.82 of total training) selected by peak eval exact-match — full 2 epochs slightly overfit on the eval split.

Data — variable-k retrieval-augmented multi-image

Built from the Chrisyichuan/screenshot-training-natural-filtered-v2 QA dataset (~104k train examples):

1. 1.For each query, retrieve top-6 screenshots from a Qwen3-VL-2B embedding index (dora-ls005 checkpoint) over 28M Wikipedia tiles.
2. 2.Per-sample, uniformly sample k ∈ {1, 2, 3, 4, 5, 6}.
3. 3.Compose the k-image set: gold always included + (k-1) non-gold hits (padded with gold repeats if retrieval has fewer than k-1 non-gold hits).
4. 4.Randomize gold position among the k images (prevents positional shortcuts).
5. 5.Apply 3x compression (each dimension scaled by 1/sqrt(3) via PIL LANCZOS).
6. 6.Train the reader with <image>×k \n {query} → gold answer.

Distribution: ~17k train samples per k value (uniform). eval/test sets likewise stratified over k∈{1..6}.

Gold-retrieval rate at top-6 across splits: ~75%. When gold is present at rank 1..6 among the retrieved hits, composition uses only retrieval results (else gold is always still included by construction).

Usage

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel
import torch

base = "Qwen/Qwen3-VL-4B-Instruct"
adapter = "Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-lora"

model = Qwen3VLForConditionalGeneration.from_pretrained(base, torch_dtype=torch.bfloat16).cuda()
model = PeftModel.from_pretrained(model, adapter).merge_and_unload()
processor = AutoProcessor.from_pretrained(base)

# k can be any integer in 1..6. Images must already be 3x-compressed.
messages = [{"role": "user", "content": [
    {"type": "image", "image": img_1},
    # ... up to img_6 ...
    {"type": "text",  "text": your_question},
]}]
# ... standard Qwen3-VL inference

Notes / limitations

• —Pixel budget is fixed at 3x. If your deployment can afford less compression, use the 2x sibling; if more, use the 4x sibling. Mixing compression at inference is untested.
• —Training always included gold in the image set. If your retriever misses the gold at inference, this adapter has not seen that distribution — expect degradation on those queries.
• —For k ∈ {1, 2, 3, 4} this adapter was evaluated with GPT-4.1 LLM-judge. k=5/6 were trained on but not explicitly benchmarked.
• —The 2× top3 oversample is asymmetric on purpose: at 3x compression the per-image budget is tight enough that purely uniform-k training under-specializes for the most common deployment k=3. Doubling top3 (104k extra fixed-k=3 samples) lifts k=3 from 0.892 to 0.932 (+0.040) while still gaining vs the uniform-k baseline at all other k. The same recipe regressed at 4x — see the 4x sibling adapter card.
• —Sister adapters at other compression levels: Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-2x-lora, Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-4x-lora.