solvrays/mdf-form-reader-phi35-vision

MDF Form Reader — Phi-3.5-Vision Fine-tuned

Vision-native handwritten insurance form understanding, fine-tuned from [microsoft/Phi-3.5-vision-instruct](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) using QLoRA.

No OCR needed. This model reads handwriting, checks checkbox states, and extracts structured data directly from scanned MDF (Monthly Disability Verification) form images.

📋 Model Summary

🚀 Quick Start

from transformers import AutoModelForCausalLM, AutoProcessor
from PIL import Image
import torch

model_id = "solvrays/mdf-form-reader-phi35-vision"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="cuda",
    trust_remote_code=True,
)

# Load your scanned MDF form image
image = Image.open("mdf_form.png").convert("RGB")

# Ask a question about the form
question = "What is the name of the physician who signed this form?"

messages = [{"role": "user", "content": f"<|image_1|>
{question}"}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=200, temperature=0.1)

answer = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(answer)

🏥 What is an MDF Form?

A Monthly Disability Verification Form (Form 441.O.MDF.O) is issued by TriPlus Services, acting as Third-Party Administrator of Penn Treaty Network America and American Network policies. It requires a licensed physician to certify a patient's ongoing disability status monthly.

Key Fields Extracted

• —Physician name, address, phone, fax
• —Submission date range (from / to)
• —Patient disability status (YES checked / NO checked)
• —Disability end date (if applicable)
• —Form completion date
• —Physician signature presence

🔬 Why Vision-Native vs OCR?

🛠️ Training Pipeline

Scanned MDF Form (PDF)
    ↓ Image pre-processing (deskew 300 DPI, bilateral denoise, CLAHE)
    ↓ Vertex AI Gemini 2.5 Flash → structured JSON annotation
    ↓ VQA triplet dataset (field extraction + OOD refusal pairs)
    ↓ Phi-3.5-Vision + QLoRA (Unsloth, 2-5× faster, 80% less VRAM)
    ↓ Merge adapters → full 16-bit model
    ↓ HuggingFace Hub (safetensors)

Training Configuration

base_model: microsoft/Phi-3.5-vision-instruct
fine_tuning_method: QLoRA (NF4, double quantization)
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.05
use_rslora: true
vision_layers: frozen
language_layers: adapted
optimizer: AdamW 8-bit (paged)
lr_scheduler: cosine
neftune_noise_alpha: 5
annotator: Vertex AI Gemini 2.5 Flash
framework: Unsloth + HuggingFace TRL

📊 Evaluation Results

OOD Refusal Rate measures how reliably the model declines to answer questions not answerable from the form (e.g. "What is the diagnosis?", "Has this claim been approved?").

⚠️ Limitations

• —Domain-specific: Trained exclusively on TriPlus Services MDF forms. Performance on other form types is not guaranteed.
• —Image quality: Works best on scans ≥ 300 DPI. Very low-resolution or heavily degraded scans may reduce accuracy.
• —Language: English only.
• —Redacted fields: Returns null for blacked-out fields (insured name/policy number).
• —Not for medical diagnosis: This model extracts administrative form data only.

📄 License

This model is released under the Apache 2.0 License. The base model (microsoft/Phi-3.5-vision-instruct) is also Apache 2.0.

🙏 Acknowledgements

• —Unsloth for 2-5× faster fine-tuning
• —Microsoft Phi-3.5-Vision for the base vision-language model
• —Vertex AI Gemini 2.5 Flash for dataset annotation
• —HuggingFace TRL for SFTTrainer