Modelpublic

kshitizjangra/qwen2vl-omr-lora-partc

sourceHugging Faceapache-2.0updated 4mo agoView on Hugging Face

0likes6downloads

Model Card

Qwen2-VL-2B LoRA — OMR Part-C `marks_obtained`

LoRA adapter on top of `Qwen/Qwen2-VL-2B-Instruct` for reading the handwritten `marks_obtained` field from cropped Part-C cells of an OMR answer sheet.

Continues training from an earlier Part-D LoRA so the adapter retains Part-D performance while picking up the Part-C marks domain.

Intended use

Single-shot OCR of a tightly cropped image containing one handwritten numeric/short value. Output is the value only, no prose.

Prompt (used at training and inference):

Read the handwritten value. Output only the value.

Training


Base model	`Qwen/Qwen2-VL-2B-Instruct`
Method	PEFT LoRA
Rank (`r`)	16
`lora_alpha`	32
`lora_dropout`	0.05
Target modules	`q_proj`, `k_proj`, `v_proj`, `o_proj`
Task type	`CAUSAL_LM`
Data	OMR Part-C cell crops, JSONL splits (80/20 train/eval)
Starting adapter	Earlier Part-D LoRA (continued fine-tune)

Usage

python

from peft import PeftModel
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
from PIL import Image
import torch

BASE = "Qwen/Qwen2-VL-2B-Instruct"
ADAPTER = "kshitizjangra/qwen2vl-omr-lora-partc"

processor = AutoProcessor.from_pretrained(BASE)
model = Qwen2VLForConditionalGeneration.from_pretrained(BASE, torch_dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

image = Image.open("crop.jpg").convert("RGB")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": image},
        {"type": "text",  "text": "Read the handwritten value. Output only the value."},
    ],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=16, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0].strip())

Files

File	Purpose
`adapter_model.safetensors`	LoRA weights
`adapter_config.json`	PEFT config
`tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`	Tokenizer + chat template
`processor_config.json`	Image/text processor

Limitations

—Trained only on Part-C marks_obtained cells. Other handwriting domains (full-page free-form, non-English script, very long sequences) are out of scope.
—Inference expects a tight crop. Loose crops or rotated images degrade accuracy.
—Same biases and limitations as the base Qwen2-VL-2B-Instruct model.

Pipeline

Source code for cropping, dataset building, training, and inference lives at: <https://github.com/kshitizjangra/omr_validator>