CoolFace
Modelpublic

shahedm2001/qwen3-VL-2B-cataract-sft-20260906-sub10

sourceHugging Faceupdated 21d agoView on Hugging Face
0likes43downloads
Model Card

Qwen3-VL-2B Cataract-Surgery SFT Adapter — 2026-09-06, 10% data (sub10)

QLoRA adapter for Qwen/Qwen3-VL-2B-Instruct, fine-tuned on cataract-surgery video (clip-level descriptions + full-video narration + CoT MCQs).

  • —Base model: Qwen/Qwen3-VL-2B-Instruct
  • —Date: 2026-09-06
  • —Data portion: 10% subset (SUBSET_RATIO=0.1) — 559 train / 689 val samples
  • —Result: train loss 2.607 → 1.213 over 70 steps, 2 epochs, exit 0

Usage

python
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

base = "Qwen/Qwen3-VL-2B-Instruct"
adapter = "shahedm2001/qwen3-VL-2B-cataract-sft-20260906-sub10"

processor = AutoProcessor.from_pretrained(adapter)  # tokenizer + processor files included
model = AutoModelForImageTextToText.from_pretrained(base, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, adapter)
model = model.merge_and_unload()  # optional: fuse to a standalone model

For video inference, pass up to 64 frames per clip (training frame count); see NFRAMES below. The repo also includes non_lora_state_dict.bin — fully-trained merger weights (the merger is full-trainable, outside LoRA).

Training curves

[image]

  • —Loss 2.607 → 1.213 (best 1.119); steady decline, mild flattening at the end — 2 epochs about right.
  • —Grad norm mean 3.06, early warmup spike to 12.0 at step 6, then stable 1.5–4 — no explosion.
  • —Cosine LR 1e-4 → 0 as configured.

Training configs (from output/logs/sft/config.txt)

GroupValues
ModelQwen/Qwen3-VL-2B-Instruct, 4-bit QLoRA (bnb_4bit_compute_dtype=bf16)
LoRAr=16, alpha=32, dropout=0.05, all LLM q/k/v/o + gate/up/down + vision linears; excludes lm_head, embed_tokens, merger, pos_embed
Frozenvision tower + LLM frozen; merger fully trainable
Batchper-device 2, grad-accum 8 → global 16; 2 epochs → 70 steps
LRLLM 1e-4, vision 2e-6, merger 1e-5; weight decay 0.1, warmup 10, cosine
VideoNFRAMES=64, pixels 98304–196608, maxseqlength 32768
Precisionbf16, tf32, flash-attn disabled (SDPA), gradient checkpointing, liger-kernel
Save/evalsave every 300 steps (keep 3); generation eval, 256 new tokens
Runwall 4h44m; GPU peak 23.9 GB / 24.6 GB, 78 °C; loss 2.607 → 1.213

Full command line is in cmd.txt (same directory as this run's logs, not uploaded).