Modelpublic

gt-free-ocr-metrics/docsim-lora

sourceHugging Faceapache-2.0updated 5mo agoView on Hugging Face

0likes14downloads

Model Card

DocSim LoRA

A LoRA-adapted CLIP+DINOv2 similarity head trained to measure visual similarity between original document page scans and their OCR reconstructions. Released as part of the OmniDocBench Render-and-Compare project.

Architecture

—Backbone A: CLIP ViT-B/32 (pretrained on LAION-2B via open_clip)
—Backbone B: DINOv2 ViT-B/14
—Projection head: two-layer MLP (hidden=512, output=256)
—LoRA adapters: rank 16, alpha 32, dropout 0.05 — applied to both CLIP and DINOv2

Training

Trained on 20,280 triplets (anchor original page, positive OCR reconstruction, negative mis-matched reconstruction) derived from the OmniDocBench Render-and-Compare dataset.

Setting	Value
Epochs	3
Batch size	16
Learning rate	1e-4
Train triplets	19,266
Val triplets	1,014
Best val accuracy	99.90%
Margin (triplet loss)	0.1

Files

File	Description
`lora_adapter_best/`	LoRA adapter at best validation accuracy
`lora_adapter_final/`	LoRA adapter at end of training
`head_state_best.pt`	Projection head weights (best checkpoint)
`head_state_final.pt`	Projection head weights (final epoch)
`config.json`	Full architecture config

Use lora_adapter_best/ + head_state_best.pt for inference.

Usage

Download via GT-free-ocr-metrics:

bash

bash download_models.sh

Then run any DocSim-based method:

bash

bash scripts/run_method.sh docsim_lora

License

Apache-2.0. The companion datasets (OmniDocBench Render-and-Compare) are CC-BY-NC-4.0.