Potestates/orena-focus-frame-qwen3.6-27b-lora

ORena FOCUS FRAME — Qwen3.6-27B LoRA adapter

LoRA adapter submitted to the FRAME track of the ORena SAVE FOCUS Challenge (MICCAI 2026): single-frame visual question answering about foreign objects in minimally invasive surgery.

Official pre-evaluation score: 59.11.

What is distinctive about it

The adapter targets 16 module names, matching 614 modules, including two families that a conventional LoRA target list misses entirely on this architecture:

• —the vision encoder (qkv, proj, linear_fc1, linear_fc2);
• —the Gated DeltaNet linear-attention layers (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj) — with full_attention_interval = 4, these are 48 of the 64 token mixers, 21 % of the model's parameters.

Under the conventional list both receive zero gradient. Adding the vision-encoder names alone was worth +3.87 points on the official metric in a matched ablation — more than a 3.4× increase in base-model size.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen3.6-27B-FP8"      # or Qwen/Qwen3.6-27B for bf16
ADAPTER = "Potestates/orena-focus-frame-qwen3.6-27b-lora"

processor = AutoProcessor.from_pretrained(BASE)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda:0")
model = PeftModel.from_pretrained(model, ADAPTER)   # do NOT merge_and_unload()
model.eval()

image = Image.open("frame.png")                     # native resolution, no resizing
messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text",  "text": "How many Clips are visible in this frame?"},
]}]
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=False)                          # required
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Three things that will otherwise cost you hours

1. 1.`enable_thinking=False` is mandatory. With the default template the model emits an extended reasoning trace and never reaches an answer.
2. 2.Do not merge the adapter into an FP8 base. It targets quantised modules; use it unmerged. The vision encoder is excluded from quantisation in the official FP8 checkpoint, so the vision-side LoRA applies losslessly either way.
3. 3.Running the FP8 base needs a `transformers` fix. quantizers_utils.should_convert_module uses re.match, which anchors only the start of a module name. The checkpoint's modules_to_not_convert contains entries ending in mlp.gate, which prefix-match mlp.gate_proj, leaving all 64 gate_proj layers unquantised and their FP8 bytes read as bf16 with scales discarded. The symptom is fluent-looking garbage; the signature is that the only UNEXPECTED keys at load are gate_proj.weight_scale_inv. Use re.fullmatch. The bf16 base is unaffected.

Training data

• —FOCUS FRAME training split — 13 748 QA rows over 92 videos (challenge-provided).
• —[`Potestates/sar-rarp50-focus-qa`](https://huggingface.co/datasets/Potestates/sar-rarp50-focus-qa) — 1 616 QA rows derived from SAR-RARP50 human segmentation masks, used for 100 steps of domain correction.

No other data. No pseudo-labels, no synthetic data, no private data.

Results

Limitations

Counting (aggregation) is the weakest capability. The model compresses the numeric range — regression slope of prediction on truth ≈ 0.51–0.59, and it never emits a value above 8. Probing indicates the true count is not recoverable from the representation, so post-hoc recalibration is capped at about +0.9 points. See the method description for the full analysis and twelve interventions that did not work.

The stage-2 domain correction is effective but blunt: it works partly by collapsing toward the majority class on out-of-distribution anatomy, and costs 2.62 points on in-distribution object recognition.

Links

• —Method description and code: https://github.com/DingTianxingjian/orena-focus-frame
• —Derived annotations: https://huggingface.co/datasets/Potestates/sar-rarp50-focus-qa
• —Challenge: https://frame.orena-focus-challenge.org/

Licence

Released under CC BY-NC-SA 4.0 for non-commercial research use. The base model is Apache-2.0; the non-commercial restriction is inherited from the challenge training data, part of which is distributed under CC BY-NC-SA.