Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-4x-lora

Qwen3-VL-4B Wiki-Screenshot QA LoRA — variable-k (1–6 images) @ 4x compression

LoRA adapter for Qwen/Qwen3-VL-4B-Instruct, fine-tuned as the reader in a retrieval-augmented Wikipedia-screenshot QA pipeline.

Unlike the earlier fixed-3-image adapters, this v2 reads any number of screenshots from 1 to 6 at inference time, at a fixed 4x pixel compression per image.

Performance (GPT-4.1 LLM-judge on 500 test examples)

Base@0x degrades sharply as the number of images grows (distractor confusion: 0.958 → 0.856 from k=1 to k=4). The 4x adapter is nearly k-invariant: 0.834 → 0.810 from k=1 to k=4.

At 4x compression, the per-image pixel budget is ~¼ the original. Even with variable-k SFT, this adapter trails the 2x and 3x variants. Use only under a hard pixel budget.

Training setup

• —Method: LoRA (r=256, alpha=256, targets=all linear layers including ViT; freeze_vision_tower=false)
• —Base model: Qwen/Qwen3-VL-4B-Instruct
• —Framework: LLaMA-Factory (fork)
• —Optimizer: cosine LR, peak 1e-5, warmup 3%
• —Effective batch: 32 (perdevice=1 × 8 GPUs × gradaccum=4)
• —cutoff_len: 6144 (scaled per compression — larger for smaller compression where each image takes more tokens)
• —Hardware: 8× H100 80GB, DeepSpeed ZeRO-2, bf16
• —Training data: vark-only (k=1..6, 104k), 2 epochs over 104k samples (final adapter at step 6502). Mixed-data and longer-epoch recipes both regressed at this compression — token-level eval improved but LLM-judge dropped, indicating the 4x pixel budget cannot absorb additional training signal.

Data — variable-k retrieval-augmented multi-image

Built from the Chrisyichuan/screenshot-training-natural-filtered-v2 QA dataset (~104k train examples):

1. 1.For each query, retrieve top-6 screenshots from a Qwen3-VL-2B embedding index (dora-ls005 checkpoint) over 28M Wikipedia tiles.
2. 2.Per-sample, uniformly sample k ∈ {1, 2, 3, 4, 5, 6}.
3. 3.Compose the k-image set: gold always included + (k-1) non-gold hits (padded with gold repeats if retrieval has fewer than k-1 non-gold hits).
4. 4.Randomize gold position among the k images (prevents positional shortcuts).
5. 5.Apply 4x compression (each dimension scaled by 1/sqrt(4) via PIL LANCZOS).
6. 6.Train the reader with <image>×k \n {query} → gold answer.

Distribution: ~17k train samples per k value (uniform). eval/test sets likewise stratified over k∈{1..6}.

Gold-retrieval rate at top-6 across splits: ~75%. When gold is present at rank 1..6 among the retrieved hits, composition uses only retrieval results (else gold is always still included by construction).

Usage

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel
import torch

base = "Qwen/Qwen3-VL-4B-Instruct"
adapter = "Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-4x-lora"

model = Qwen3VLForConditionalGeneration.from_pretrained(base, torch_dtype=torch.bfloat16).cuda()
model = PeftModel.from_pretrained(model, adapter).merge_and_unload()
processor = AutoProcessor.from_pretrained(base)

# k can be any integer in 1..6. Images must already be 4x-compressed.
messages = [{"role": "user", "content": [
    {"type": "image", "image": img_1},
    # ... up to img_6 ...
    {"type": "text",  "text": your_question},
]}]
# ... standard Qwen3-VL inference

Notes / limitations

• —Pixel budget is fixed at 4x. If your deployment can afford less compression, use the 2x sibling; if more, use the 4x sibling. Mixing compression at inference is untested.
• —Training always included gold in the image set. If your retriever misses the gold at inference, this adapter has not seen that distribution — expect degradation on those queries.
• —For k ∈ {1, 2, 3, 4} this adapter was evaluated with GPT-4.1 LLM-judge. k=5/6 were trained on but not explicitly benchmarked.
• —v1 of this adapter (trained on fixed k=3 only) achieved slightly higher k=3 score but collapses at other k. This v2 trades ~0.02 at k=3 for flexibility across k=1..6.
• —Sister adapters at other compression levels: Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-2x-lora, Chrisyichuan/qwen3vl-4b-wiki-screenshot-multik-3x-lora.