CoolFace
Modelpublic

TakTak/qwen3-vl-8b-distill32-image-math-lora

sourceHugging Facecc-by-nc-4.0updated 1mo agoView on Hugging Face
0likes13downloads
Model Card

Qwen3-VL-8B Distill32 Image-Math LoRA

Release role

This is the primary validated adapter in the accompanying two-adapter research release. It is a bounded teacher-student experiment, not a promoted general-purpose math specialist or a production tutoring model.

Qwen/Qwen3-VL-32B-Thinking generated Korean candidate solutions for a multimodal MMK12 subset. Only answer-verified traces were retained, and a LoRA adapter was trained on Qwen/Qwen3-VL-8B-Thinking with the vision tower frozen.

Model details

  • —Base model: Qwen/Qwen3-VL-8B-Thinking
  • —Base revision: 92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b
  • —Teacher model: Qwen/Qwen3-VL-32B-Thinking
  • —Teacher revision: 7edd10ffd1196091948fb245ff63e406ccb2d4d1
  • —Adapter: PEFT LoRA, rank 32, alpha 64, dropout 0.05
  • —Target modules: q/k/v/o and gate/up/down projections
  • —Vision tower: frozen
  • —Adapter weight SHA-256: d3a64c058fe6a1449b9842a18e49cc4d0f19b59f323723c2d683ac57e82b8a18

Training data and filtering

The source pool was the math subset of FanqingM/MMK12. A deterministic 1,200-row sample was submitted to the teacher. Candidate traces were retained only when the extracted final answer matched the source answer and the trace stayed below the configured length cap.

  • —Accepted rows: 755
  • —Training rows: 675
  • —Validation rows: 80
  • —Images per retained row: exactly 1
  • —Training JSONL SHA-256: e7a1aa80238a1fb8198b4e7c66c8601be5911bce0ce15ded33734f1644909e8d
  • —Validation JSONL SHA-256: 0c35a42e259ac262baa2ba1e953b5138a28eb6f5b4bbf8adb1ef6ad0c6f024a0

Training rows and images are not redistributed. Obtain MMK12 from its source repository and comply with its license and attribution requirements.

Training procedure

  • —Epochs: 2
  • —Learning rate: 1e-4
  • —Precision: BF16
  • —Per-device batch size: 1
  • —Gradient accumulation: 8
  • —Maximum sequence length: 10,240
  • —Maximum image long side: 896 pixels
  • —Seed: 1234
  • —Hardware: 8 GPUs

Evaluation

Base and adapter arms used identical inputs, prompts, greedy decoding, token budgets, scorer, and UID-aligned paired comparisons.

Evaluation setNBaseAdapterDeltaInterpretation
Original AIHub15 head subset50065.4%72.0%+6.6ppLeakage-clean bounded positive; family-wise significant
New pool-representative subset50054.6%56.4%+1.8ppDirectional, not significant; p=0.397
New pool subset4,00053.5%55.475%+1.975ppSignificant; p=0.002353
Pre-registered combined primary4,50053.622%55.578%+1.956ppSignificant; p=0.001451; 95% CI [+0.756, +3.156]pp

The pre-registered practical threshold was +2.0pp. The combined point estimate is narrowly below that threshold, while its confidence interval crosses it. The defensible conclusion is a small, statistically significant, pool-level effect whose magnitude is smaller than the original head-subset result.

The adapter used fewer output tokens and reached the token cap less often than Base in the combined evaluation, so the gain was not produced by spending a larger output budget.

Intended use

  • —Reproduce or inspect a bounded teacher-trace distillation adapter.
  • —Study controlled paired evaluation of multimodal LoRA adapters.
  • —Use as one candidate in category- or answer-type-aware model selection research, with Base as a fallback.

Limitations

  • —Do not claim broad Korean math or science improvement.
  • —Do not claim a completed specialist, learned router, deployed ensemble, or production system.
  • —Do not generalize the original +6.6pp head-subset magnitude to the full AIHub15 pool.
  • —MMK12 transfer evaluation included a small overlap caveat and is not clean independent transfer evidence.
  • —Training data were not Korean-native; the teacher was prompted to answer in Korean.
  • —Safety, fairness, and high-stakes educational use were not evaluated.

Loading

python
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel

base_id = "Qwen/Qwen3-VL-8B-Thinking"
base_revision = "92f3c4b4feadd3a016ef468d103bb5f58b2a2c6b"
adapter_id = "TakTak/qwen3-vl-8b-distill32-image-math-lora"

processor = AutoProcessor.from_pretrained(base_id, revision=base_revision)
base = AutoModelForImageTextToText.from_pretrained(
    base_id,
    revision=base_revision,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)

License and upstream sources

The adapter is released under CC BY-NC 4.0. The pinned Qwen base and teacher revisions are Apache-2.0. MMK12's pinned source snapshot contains an Apache-2.0 license file. No raw source data, evaluation rows, predictions, or scored row-level outputs are included.

  • —Base: https://huggingface.co/Qwen/Qwen3-VL-8B-Thinking
  • —Teacher: https://huggingface.co/Qwen/Qwen3-VL-32B-Thinking
  • —Data source: https://huggingface.co/datasets/FanqingM/MMK12