solvrays/mdf-form-reader-phi35-vision

1---2language:3  - en4license: apache-2.05library_name: transformers6tags:7  - vision-language-model8  - document-understanding9  - handwritten-text10  - insurance-forms11  - vqa12  - phi-3.5-vision13  - lora14  - qlora15  - unsloth16  - medical-forms17  - ocr-free18pipeline_tag: image-to-text19base_model: microsoft/Phi-3.5-vision-instruct20datasets:21  - custom-mdf-forms22metrics:23  - exact_match24model-index:25  - name: mdf-form-reader-phi35-vision26    results:27      - task:28          type: visual-question-answering29          name: Visual Question Answering (MDF Forms)30        metrics:31          - type: exact_match32            value: 033            name: Exact Match (%)34          - type: ood_refusal_rate35            value: 036            name: OOD Refusal Rate (%)37---38 39# MDF Form Reader — Phi-3.5-Vision Fine-tuned40 41**Vision-native handwritten insurance form understanding, fine-tuned from [microsoft/Phi-3.5-vision-instruct](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) using QLoRA.**42 43> **No OCR needed.** This model reads handwriting, checks checkbox states, and extracts structured data directly from scanned MDF (Monthly Disability Verification) form images.44 45---46 47## 📋 Model Summary48 49| Property | Value |50|---|---|51| **Base Model** | `microsoft/Phi-3.5-vision-instruct` (4.2B) |52| **Task** | Visual Question Answering on MDF forms |53| **Fine-tuning Method** | QLoRA (r=16, alpha=32) via Unsloth |54| **Quantization** | 4-bit NF4 (training) → 16-bit merged |55| **Annotator** | Vertex AI Gemini 2.5 Flash |56| **Exact Match** | 0% |57| **OOD Refusal Rate** | 0% |58| **License** | Apache 2.0 |59 60---61 62## 🚀 Quick Start63 64```python65from transformers import AutoModelForCausalLM, AutoProcessor66from PIL import Image67import torch68 69model_id = "solvrays/mdf-form-reader-phi35-vision"70 71processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)72model = AutoModelForCausalLM.from_pretrained(73    model_id,74    torch_dtype=torch.bfloat16,75    device_map="cuda",76    trust_remote_code=True,77)78 79# Load your scanned MDF form image80image = Image.open("mdf_form.png").convert("RGB")81 82# Ask a question about the form83question = "What is the name of the physician who signed this form?"84 85messages = [{"role": "user", "content": f"<|image_1|>86{question}"}]87text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)88 89inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")90 91with torch.no_grad():92    out = model.generate(**inputs, max_new_tokens=200, temperature=0.1)93 94answer = processor.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)95print(answer)96```97 98---99 100## 🏥 What is an MDF Form?101 102A **Monthly Disability Verification Form (Form 441.O.MDF.O)** is issued by TriPlus Services, acting as Third-Party Administrator of Penn Treaty Network America and American Network policies. It requires a licensed physician to certify a patient's ongoing disability status monthly.103 104### Key Fields Extracted105 106- Physician name, address, phone, fax107- Submission date range (from / to)108- Patient disability status (YES checked / NO checked)109- Disability end date (if applicable)110- Form completion date111- Physician signature presence112 113---114 115## 🔬 Why Vision-Native vs OCR?116 117| Challenge | OCR Approach | This Model |118|---|---|---|119| Cursive physician names | Fails ("Carnazzo", "Kruszka") | Reads directly from image |120| Checkbox state (YES/NO) | Misses (no text to extract) | Sees the ✓/✗ mark in context |121| Date grid cells (MM/DD/YYYY) | Digit confusion in small boxes | Layout-aware reading |122| Signature field | Garbage output | Correctly ignored |123| Handwritten addresses | High error rate | Contextual correction |124 125---126 127## 🛠️ Training Pipeline128 129```130Scanned MDF Form (PDF)131    ↓ Image pre-processing (deskew 300 DPI, bilateral denoise, CLAHE)132    ↓ Vertex AI Gemini 2.5 Flash → structured JSON annotation133    ↓ VQA triplet dataset (field extraction + OOD refusal pairs)134    ↓ Phi-3.5-Vision + QLoRA (Unsloth, 2-5× faster, 80% less VRAM)135    ↓ Merge adapters → full 16-bit model136    ↓ HuggingFace Hub (safetensors)137```138 139### Training Configuration140 141```yaml142base_model: microsoft/Phi-3.5-vision-instruct143fine_tuning_method: QLoRA (NF4, double quantization)144lora_rank: 16145lora_alpha: 32146lora_dropout: 0.05147use_rslora: true148vision_layers: frozen149language_layers: adapted150optimizer: AdamW 8-bit (paged)151lr_scheduler: cosine152neftune_noise_alpha: 5153annotator: Vertex AI Gemini 2.5 Flash154framework: Unsloth + HuggingFace TRL155```156 157---158 159## 📊 Evaluation Results160 161| Metric | Value |162|---|---|163| Exact Match (field extraction) | 0% |164| OOD Refusal Rate | 0% |165| Evaluation Set | Held-out MDF form pages |166 167**OOD Refusal Rate** measures how reliably the model declines to answer questions not answerable from the form (e.g. "What is the diagnosis?", "Has this claim been approved?").168 169---170 171## ⚠️ Limitations172 173- **Domain-specific**: Trained exclusively on TriPlus Services MDF forms. Performance on other form types is not guaranteed.174- **Image quality**: Works best on scans ≥ 300 DPI. Very low-resolution or heavily degraded scans may reduce accuracy.175- **Language**: English only.176- **Redacted fields**: Returns `null` for blacked-out fields (insured name/policy number).177- **Not for medical diagnosis**: This model extracts administrative form data only.178 179---180 181## 📄 License182 183This model is released under the **Apache 2.0 License**.184The base model ([microsoft/Phi-3.5-vision-instruct](https://huggingface.co/microsoft/Phi-3.5-vision-instruct)) is also Apache 2.0.185 186---187 188## 🙏 Acknowledgements189 190- [Unsloth](https://github.com/unslothai/unsloth) for 2-5× faster fine-tuning191- [Microsoft Phi-3.5-Vision](https://huggingface.co/microsoft/Phi-3.5-vision-instruct) for the base vision-language model192- [Vertex AI Gemini 2.5 Flash](https://cloud.google.com/vertex-ai) for dataset annotation193- [HuggingFace TRL](https://github.com/huggingface/trl) for SFTTrainer194