ericssonbear/qwen3-vl-emb-8b-mrright-zhtw-lora

Qwen3-VL-Embedding-8B · Mr. Right zh-TW retrieval LoRA

Two LoRA adapters for Qwen/Qwen3-VL-Embedding-8B that turn it into a Traditional-Chinese multimodal document retriever over the Mr. Right zh-TW corpus (769,245 Wikipedia-derived image+text documents).

On Traditional-Chinese queries the pair lifts the backbone from R@1 .5459 → .7283 and R@10 .8134 → .9052 (test, clean protocol, n = 1,951), for 43.6 M trainable parameters — no backbone weights are redistributed.

⚠️ The deployment is asymmetric — you need both adapters

Query and document vectors only live in the same space if each side uses its own adapter. Using query_encoder for documents (or vice versa) will silently degrade retrieval. If you use the pre-computed embeddings, they were produced by doc_encoder and you only need query_encoder at query time.

Usage

import torch
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

BASE = "Qwen/Qwen3-VL-Embedding-8B"
REPO = "ericssonbear/qwen3-vl-emb-8b-mrright-zhtw-lora"

# NOTE: load the ...ForConditionalGeneration class, NOT AutoModel.
# The checkpoint tensors are prefixed `model.`; AutoModel resolves to the bare
# Qwen3VLModel, whose keys do not match -- it silently random-initialises every
# weight and the LoRA keys do not match either. Always check loading info.
processor = AutoProcessor.from_pretrained(BASE, max_pixels=1024 * 28 * 28)
full, info = Qwen3VLForConditionalGeneration.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda", output_loading_info=True)
assert not [k for k in info["missing_keys"] if k != "lm_head.weight"]
assert not info["unexpected_keys"]

PeftModel.from_pretrained(full, REPO, subfolder="query_encoder", is_trainable=False)
full.eval()
base = full.model                    # inner module -> last_hidden_state

QUERY_INSTRUCTION = "給定一個圖文混合查詢，檢索相關的維基百科圖文文檔："

def encode(texts):
    prompts = [processor.apply_chat_template(
        [{"role": "user", "content": f"指令：{QUERY_INSTRUCTION}\n查詢：{t}"}],
        tokenize=False, add_generation_prompt=False) for t in texts]
    inp = processor(text=prompts, return_tensors="pt", padding=True,
                    truncation=True, max_length=1024).to("cuda")
    with torch.inference_mode():
        h = base(**inp, return_dict=True).last_hidden_state
    last = inp["attention_mask"].sum(dim=1) - 1          # last-token pooling
    v = h[torch.arange(h.size(0), device=h.device), last].float()
    return torch.nn.functional.normalize(v, p=2, dim=-1)

q = encode(["一座建於一八九一年、位於康涅狄格州新倫敦、並作為中學使用的大型磚造建築。"])

Retrieval is cosine similarity (both sides are L2-normalised) — a plain dot product.

Instructions matter (the model is instruction-aware). Query side, by query type:

Document side always uses 為以下文檔產生向量表示，以用於檢索任務：, with the image passed as an image content block (896×896 pixel cap) followed by title_zhtw\ndoc_text_zhtw.

Results

Test split, clean protocol (gold document present in the corpus, n = 1,951), query_multi used as plain text — see Evaluation caveats.

Against retrieval baselines (Traditional-Chinese queries)

Every baseline sees the image too: documents are indexed with their generated Traditional-Chinese image description, so BM25 and e5 are multimodal-by-proxy rather than text-only strawmen.

Training ladder (Traditional-Chinese queries)

Cumulative: R@1 +18.24 pp, R@10 +9.18 pp, MRR +15.71 pp over the zero-shot backbone. The three biggest levers are mt-mix query training (+5.17 pp R@1), the initial instruction LoRA (+4.20), and unfreezing the document tower (+4.10).

Cross-lingual

English queries are not worse than Chinese — the adapters are cross-lingually symmetric over a bilingual document bank.

Does the image actually matter?

Single-factor ablation, same adapters, only the document modality changes:

Removing the image costs −11.7 pp R@1 / −7.3 pp R@10 (McNemar exact and 20k paired bootstrap, all p < .0001). The image-only bank still reaches R@10 .32 out of 769,245 candidates (chance ≈ 1e-5), so the two modalities carry independent, complementary signal.

What did not work (reported deliberately)

• —BM25 fusion is a net negative under r5. RRF with BM25 costs −3.4 pp MRR (p = .0001). The deployed system is a single dense leg with no lexical index. This was not true at earlier rungs — BM25 helped until the retriever got strong enough.
• —Explicit margin loss: zero gain. With the full 769k corpus in the InfoNCE denominator at temperature 0.03, the objective already concentrates gradient on the hardest negatives; static hard-negative mining is redundant.
• —LLM-judged hard positives: zero gain. Re-training on human/judge-reviewed pairs (embedder_13_h) did not beat r5 and was not deployed.

If you are reproducing this recipe, these three cost real compute and bought nothing.

Evaluation caveats

• —The `query_multi` image trap. The upstream mixed query ships with a query image, and 1,992 of ~2,000 test queries use the same image file as their gold document. An untrained backbone scores R@10 = .952 on it — that is image hashing, not retrieval. All numbers here use query_multi as plain text, with no query image.
• —Clean protocol. 96 of 2,047 test queries have a gold document that is absent from the corpus (dead image URL), imposing a recall ceiling of .953 on the full set. Numbers are on the clean subset, n = 1,951.
• —Comparing against numbers computed under either different convention is meaningless.

Offline ceiling (not deployed)

Adding a training-free late-interaction (MaxSim) rerank over the top-20, scored with doc_encoder token states, reaches R@1 .7719 / R@10 .9257 / MRR .8237. It roughly doubles query latency and was left out of the deployment.

Training setup

LoRA rank 16, alpha 32, dropout 0.05, on the language-model tower only (self_attn.{q,k,v,o}_proj and mlp.{gate,up,down}_proj for all 36 layers) — the vision tower and merger are untouched. 504 LoRA tensors, 43.6 M parameters. Objective: InfoNCE at temperature 0.03 with the full 769,245-document bank as negatives, Matryoshka losses at 4096/2048/1024/512/256. Trained on the NCHC 晶創 (Nano5) cluster. Base weights are bf16; adapters are fp32.

Training data, all from `ericssonbear/mr-right-zhtw`: the 900 human-annotated pairs in queries-finetune (rung 1, using the *_zhtw Gemma 4 translation — not the *_zhtw_hv human-verified one), then the pseudo-pairs-zh and pseudo-pairs-hard configs for the later rungs. The embedder_13_h null result above is exactly the experiment of swapping in the human-verified translations.

Limitations and intended use

• —Tuned for retrieval over this Wikipedia-derived corpus in Traditional Chinese and English. It is not a general-purpose sentence embedder and has not been evaluated on MTEB or on any other domain.
• —Evaluation queries are English queries written by human annotators, then translated to Traditional Chinese and individually human-verified. They are still not queries a Taiwanese speaker wrote from scratch, so they carry translationese: there is no natively authored Traditional-Chinese evaluation slice, and the zh numbers should be read as an upper bound on what native-speaker queries would give.
• —The document side of the corpus is bulk machine translation (Gemma 4) with no human review, so document-side translation errors are baked into the bank these adapters were trained on.
• —No adversarial, fairness, or domain-shift testing was done beyond the cross-lingual symmetry check above.
• —Document vectors were computed with a 896×896 pixel cap and max_length 1024 text; matching those settings matters if you re-encode documents yourself.
• —Inherits the biases of Wikipedia, of the backbone, and of the VLM that wrote the captions the documents were indexed with.

License

Adapter weights: Apache-2.0, matching the Qwen/Qwen3-VL-Embedding-8B backbone. Backbone weights are not redistributed here. The training and evaluation data are CC BY-SA 4.0 (see the dataset card).

Citation

@misc{mrright_zhtw_lora_2026,
  title  = {Qwen3-VL-Embedding-8B Mr. Right zh-TW retrieval LoRA},
  author = {Hsin-Yu Lin and Hong-Yan Huang and Shau-Yung Hsu and Song-Duo Ma and Pin-Yu Chen and Chu-Yun Chen and Wei-Te Ho and Pu-Jen Cheng},
  year   = {2026},
  url    = {https://huggingface.co/ericssonbear/qwen3-vl-emb-8b-mrright-zhtw-lora}
}

Built by Hsin-Yu Lin, Hong-Yan Huang, Shau-Yung Hsu, Song-Duo Ma, Pin-Yu Chen, Chu-Yun Chen, Wei-Te Ho and Pu-Jen Cheng. Funded under National Science and Technology Council, Taiwan, project NSTC 115-2634-F-001-006 ("Advancing Next-Generation Frontier AI Research" programme). Compute provided by the NCHC 晶創 (Nano5) cluster.