Modelpublic

ppokhrel2109/freezeshift-dual-vlm

sourceHugging Faceotherupdated 1mo agoView on Hugging Face

0likes9downloads

Model Card

FreezeShift Dual — Bounded Encoder Adaptation

A bounded-adaptation image-text retrieval model: 49.2M deployed parameters, of which 4.9M are trainable.

Research artifact. Not licensed for commercial use. See NOTICE.

Code: https://github.com/Pranav210901/Efficient-VLM

What this is

A frozen DINOv3 ViT-S/16 vision tower (224 px) and a frozen all-MiniLM-L6-v2 text tower, joined by learned token aggregation and residual projections into a shared normalised 384-d space. MobileCLIP2-S0 provided a distillation signal during training only and is not part of this model.

Identical to M_T1 plus rank-128 LoRA on the attention projections of the final four blocks of both towers. This is the study's upper bound on what limited encoder adaptation recovers: +9.3pp test R@1 over the fully frozen model for 2.0M extra trainable parameters.


Deployed parameters	49,162,629
Trainable at inference	4,862,469 (9.9%)
Seed	44
Embedding dim	384
Image size	224

Results

Flickr30k Karpathy test, opened once after the model was frozen, with no training of any kind. This checkpoint (seed 44) scores 62.04% mean bidirectional R@1.

Three-seed means for context:

Model	Flickr30k test R@1	Params
M_T1 (fully frozen)	52.90 ± 0.87	47.2M
FreezeShift dual	62.20 ± 0.45	49.2M
OpenCLIP ViT-B/32	68.22	151.3M
MobileCLIP2-S0	78.25	74.8M
SigLIP2 ViT-B/32	80.46	376.9M

This model does not beat the compact references. MobileCLIP2-S0 is smaller than OpenCLIP and scores higher than both models released here. The contribution is a measured boundary on frozen-encoder alignment under a fixed latency budget, not a state-of-the-art retrieval model.

Zero-shot classification transfer is weak (CIFAR-100 ~36-38%, Oxford-IIIT Pet ~8-10%, EuroSAT ~22-24%) and should be treated as a negative result.

Seed selection

The seed was chosen on the validation split, never on test. Selecting on test would have picked a different seed and inflated the reported figure.

Checkpoint provenance

This exports the COCO-dev-selected epoch. The headline validation figures in the dissertation (54.487% / 63.416%) use Flickr-validation epoch selection; under the deployed selection the same models score 54.36% / 63.04% on validation. The test figures above are unaffected by this distinction.

Files

file	contents
`model.safetensors`	full stack, frozen towers included — self-contained
`adapter.safetensors`	trainable parameters only; frozen towers fetched at load
`config.json`	frozen architecture and data config
`training_recipe.json`	training recipe as run
`fingerprint.json`	provenance hashes (code, config, split, digest)
`load_model.py`	loader for both variants

The adapter-only variant contains no third-party weights, which avoids redistributing DINOv3 entirely if that matters for your use.

Usage

python

# requires the repo code on PYTHONPATH
from load_model import load

model = load(".", adapter_only=False)          # or True for the adapter variant
image_embeds = model.encode_image(pixel_values) # (B, 384), L2-normalised
text_embeds  = model.encode_text(["a dog on a beach"])

Licensing

Component	Terms
DINOv3 ViT-S/16	Meta DINOv3 License — copy included as `LICENSE-DINOv3.md`
all-MiniLM-L6-v2	Apache-2.0
MobileCLIP2-S0 (teacher, training only)	Apple ML Research Model License — research only

The DINOv3 License requires that derivatives be distributed under its terms with a copy of the Agreement provided, that publications acknowledge DINO Materials, and that use comply with Trade Controls. It prohibits military, weapons, espionage, and nuclear applications. MobileCLIP2-S0 weights are not redistributed here.

Citation

Pokhrel, P. Efficient Frozen Vision-Language Alignment. MSc dissertation, University of Surrey, 2026.