sivakorn-su/typhoon-ocr-7b-thai-handwriting-lora-v1

Typhoon OCR 7B — Thai Handwriting LoRA

LoRA adapter on typhoon-ai/typhoon-ocr-7b, fine-tuned to read Thai handwriting from an image and return the transcription as JSON ({"natural_text": "..."}).

Overview

• —Base model: typhoon-ai/typhoon-ocr-7b (Qwen2.5-VL-7B, Apache-2.0)
• —Task: Thai handwriting image → text
• —Type: PEFT/LoRA adapter — load on top of the base model (base weights are not redistributed here)
• —Intended inference: images at 1024px (max_pixels=1048576), greedy decoding (do_sample=False), max_new_tokens=512. Other regimes are outside the evaluation scope.

Performance

This is the evaluation attached to the branch promoted to main: phase2-downsample-money-keep25-300steps, evaluated on the 559-sample CPE-OPH full test with 1024px images, greedy decoding, and max_new_tokens=512.

Baseline: phase2-baseline-greedy-res1024. Source: published metrics artifact.

The CPE-OPH full-test result is not a clean generalization score because gold texts overlap between its train and test splits.

Example use

Install dependencies first (qwen-vl-utils is a separate package — hyphens in the pip name, underscores in the import):

pip install "transformers>=4.49" accelerate peft torch pillow qwen-vl-utils

import json, torch
from PIL import Image
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from peft import PeftModel
from qwen_vl_utils import process_vision_info

BASE = "typhoon-ai/typhoon-ocr-7b"
ADAPTER = "sivakorn-su/typhoon-ocr-7b-thai-handwriting-lora-v1"  # the `main` adapter

processor = AutoProcessor.from_pretrained(BASE, max_pixels=1048576)  # 1024px regime
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(BASE, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

PROMPT = (
    "Below is an image of a document page along with its dimensions. "
    "Simply return the markdown representation of this document, presenting tables in markdown format as they naturally appear.\n"
    "If the document contains images, use a placeholder like dummy.png for each image.\n"
    "Your final output must be in JSON format with a single key `natural_text` containing the response.\n"
    "RAW_TEXT_START\n\nRAW_TEXT_END"
)

image = Image.open("handwriting.png").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "text", "text": PROMPT},
    {"type": "image", "image": image},
]}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, videos=video_inputs,
                   padding=True, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)  # greedy
decoded = processor.batch_decode(out[:, inputs.input_ids.shape[1]:],
                                 skip_special_tokens=True)[0].strip()
print(json.loads(decoded)["natural_text"])

Prompt note (important): the prompt above is the exact one this adapter was fine-tuned with — Typhoon's default task template with an empty `RAW_TEXT` anchor and no injected page dimensions. Base Typhoon normally runs via the typhoon-ocr package (ocr_document), which builds a richer prompt (OCR anchor text from get_anchor_text + real dimensions). Because this adapter was trained on the simplified, empty-anchor prompt, use it as-is — do not substitute the package's full prompt, or inference will mismatch training. (Typhoon: the model only works with its specific prompt templates.)

Limitations

The Performance table reports only the CPE-OPH evaluation used to promote this adapter to main. Phase 3 evaluations belong to their experimental branches and are intentionally excluded here. The result does not establish performance on full document forms, layouts, unseen writers, or capture conditions outside the evaluation set.

Lineage

Promoted to main from branch `phase2-downsample-money-keep25-300steps` after full-test evaluation and promotion-gate checks. Base: typhoon-ai/typhoon-ocr-7b; trained and evaluated on Thinnaphat/TH-HANDWRITTEN-CPE-OPH2025.

License & attribution

Apache-2.0, inheriting the base model typhoon-ai/typhoon-ocr-7b → Qwen/Qwen2.5-VL-7B-Instruct. Training data: Thinnaphat/TH-HANDWRITTEN-CPE-OPH2025 (a subset of iapp/thai_handwriting_dataset, Apache-2.0). This repository is a LoRA adapter only.