jhanwarsid/qwen3.5-2b-omr-lora
Qwen3.5-2B OMR Field OCR (LoRA, categorized bad-fields)
A LoRA fine-tune of [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) (a vision-language model) that reads handwritten values from cropped fields of student OMR sheets. Given a single cropped field image, it outputs the transcribed value — or a category token when the field is not a normal value.
Supported fields
Bad-field categories
Instead of guessing on unreadable fields, the model emits a category token:
<blank>— the field is empty (nothing written)<strikethrough>— a value was written and then crossed out
(<unclear> is described in the prompt convention but had no training examples in this release, so the model is not expected to emit it.)
Treat any category token as "flag for manual review".
Evaluation
Exact-match accuracy on a held-out validation set (702 examples, 10% stratified split of the combined data):
Bad-field detection: precision 1.00, recall 0.80 (bad-field examples are rare, so recall is measured on a small sample). Remaining value errors are single ambiguous-digit confusions (e.g. 0↔9, 7↔1, 6↔8).
Usage
import torch
from PIL import Image
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor
BASE = "Qwen/Qwen3.5-2B"
ADAPTER = "jhanwarsid/qwen3.5-2b-omr-lora"
CATEGORIES = {"<blank>", "<strikethrough>", "<unclear>"} # outputs meaning "review"
PROMPTS = {
"registration_no": "This image is a single field cropped from a student OMR sheet containing a handwritten registration number (digits and letters, e.g. 2572U00739). Transcribe it exactly as written, outputting only the value with no spaces, quotes, labels, or explanation. If the field is empty with nothing written, output <blank>. If a value was written and then struck through or crossed out, output <strikethrough>.",
"roll_no": "This image is a single field cropped from a student OMR sheet containing a handwritten roll number (digits only, e.g. 7200739). Transcribe it exactly as written, outputting only the value with no spaces, quotes, labels, or explanation. If the field is empty with nothing written, output <blank>. If a value was written and then struck through or crossed out, output <strikethrough>.",
"course_code": "This image is a single field cropped from a student OMR sheet containing a handwritten course code (a letter followed by digits, e.g. U12028). Transcribe it exactly as written, outputting only the value with no spaces, quotes, labels, or explanation. If the field is empty with nothing written, output <blank>. If a value was written and then struck through or crossed out, output <strikethrough>.",
"marks_obtained": "This image is a single field cropped from a student OMR sheet containing the handwritten marks obtained (a number, e.g. 45). Transcribe it exactly as written, outputting only the value with no spaces, quotes, labels, or explanation. If the field is empty with nothing written, output <blank>. If a value was written and then struck through or crossed out, output <strikethrough>.",
}
processor = AutoProcessor.from_pretrained(BASE, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
BASE, dtype=torch.bfloat16, trust_remote_code=True
)
model = PeftModel.from_pretrained(model, ADAPTER).eval()
# model = model.to("cuda") # or "mps"
def read_field(image_path: str, field: str, max_new_tokens: int = 24) -> str:
img = Image.open(image_path).convert("RGB")
messages = [{
"role": "user",
"content": [
{"type": "image", "image": img},
{"type": "text", "text": PROMPTS[field]},
],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[[img]], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=False)
value = processor.tokenizer.decode(
out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True
).strip()
return value # a value, or a category token in CATEGORIES (-> manual review)
print(read_field("roll_no.jpg", "roll_no"))Training details
- Base: Qwen/Qwen3.5-2B (vision-language)
- Method: LoRA (
r=16,alpha=32, dropout 0.05,target_modules="all-linear") - Objective: supervised fine-tuning; loss computed only on the answer tokens
- Data: ~7k labeled cropped OMR field images across the four fields, including bad images labeled
<blank>/<strikethrough> - Precision: bf16, 2 epochs
Intended use & limitations
Built for transcribing the four OMR fields above from pre-cropped field images. It is not a general OCR model and expects a single field per image. Trained on one institution's OMR layout/handwriting, so accuracy may drop on different forms. Bad-field examples were rare in training, so always keep the category tokens as a manual-review path. Inherits the license and usage terms of the base model (Qwen/Qwen3.5-2B).