AbstractPhil/clip-vitb-mini-distilled

clip-vitb-mini-distilled

An 8.66M-parameter ViT image encoder (10.0% of a CLIP-B/16 image tower) producing 512-d embeddings compatible with the CLIP-B/16 LAION-2B text tower. The primary checkpoint was distilled on CC12M (10,968,539 images) against the generalized-Procrustes consensus of five CLIP teachers — never against the deployment teacher — and carries a frozen 512×512 rotation that maps its outputs into the deployment frame, where it outperforms the student distilled directly against that teacher on every task gauge, both seeds (full tables below).

Quick start (AutoModel)

import torch
from transformers import AutoModel, AutoImageProcessor

repo = "AbstractPhil/clip-vitb-mini-distilled"
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained(repo)

px = proc(images=[img], return_tensors="pt")["pixel_values"]
emb = model.get_image_features(px)     # (1, 512), L2-normalized,
                                       # deployment (LAION-B/16) frame

The rotation is applied by default (config.apply_rotation); pass apply_rotation=False to get_image_features for the raw consensus-frame embedding. Weights are safetensors; the modeling code is in this repo (modeling_clip_mini.py, configuration_clip_mini.py).

Why a rotation exists at all — and when it doesn't. This model's consensus target is a generalized-Procrustes running mean of five teachers, which has no privileged frame: the student lands in the consensus frame rather than any teacher's, so a fitted rotation carries it the rest of the way. That is a property of how the target was built, not of consensus distillation. A sibling campaign built its consensus by mapping every teacher into the frame of one reference member of the set; its frame gap fell monotonically to ~.001 during training and it ships with no rotation. If you build your own consensus target, pick a reference member for a pinned frame or a mean for symmetry — and know which you chose.

Evaluation-path note: all published numbers were measured with the tensor-bicubic evaluation pipeline (interpolate to 160px, CLIP norm — the path implemented in loader.py and train/dist_bed.py). On CIFAR-scale (32px) inputs the shipped CLIPImageProcessor path yields 0.5545 vs 0.5708 zero-shot for the same weights, a resize-path difference that matters only for very small source images; verify against ledger numbers with the tensor path.

Repository inventory

Architecture

Results

Headline — champion (rotated) vs the directly-distilled student, CC12M, seed 0 / seed 1:

Champion capability: 44.7% / 48.1% of the teacher's zero-shot mean (CIFAR-10+100) at 10% of its image-tower parameters. Scaling arc across data generations: 21.2% (118k images) → 42.0/41.5% (11M, direct InfoNCE) → 44.7/48.1% (11M, consensus + rotation).

Full CC12M matrix (nine objectives × two seeds, 88k steps ≈ 2.05 epochs; agree = cosine to deployment teacher unless (own) = the arm's own target):

Rotation forensics (agree→deployment pre → post; zs C10 pre → post; both seeds):

Note the pure-MSE consensus row: rotated agreement .765/.763 exceeds direct mimicry's in-frame .735/.738 — the consensus target carries the deployment teacher's geometry better than direct distillation preserves it — but without a contrastive term it cannot retrieve (rotated R@1 .046/.051). The composite's two terms are both load-bearing.

Generation one (COCO, 118k images, 8k steps), for the scale comparison: mimicry .2693/.2657 zs vs InfoNCE .2668/.2308 (mimicry won coarse zero-shot at small scale; the verdict reversed at 11M); InfoNCE retrieval .0580/.0544 vs .0248/.0262; consensus raw .1543/.1609 — its loss to single-teacher distillation was preregistered and refuted 2/2 at this scale before reversing at CC12M with the rotation. Full generation-one ledgers included.

Training

Everything needed to reproduce any checkpoint is in train/ (train/TRAINING.md is the step-by-step guide). Summary:

Evaluation protocol

CIFAR-10/100 zero-shot via the LAION-B/16 text tower ("a photo of a {class}"), images tensor-upscaled (bicubic) to the student's 160px input; teacher judged at its native 224px in the same harness; floor (random init) and ceiling (teacher) bracket every table. COCO retrieval: image→text R@1/R@5 over the 5,000 val images against first-caption text embeddings, paired protocol. Agreement: mean cosine to the arm's target embeddings on COCO val. Effective rank and a spread statistic (CV of Cayley-Menger 4-simplex volumes on a fixed 16-d projection) are read-only health instruments. Every evaluation is deterministic: --eval-only re-runs reproduce ledger rows to four decimal places, and this release was gated by fresh tokenless downloads reproducing the headline numbers exactly (raw-loader path and AutoModel path, bit-parity verified).

Limitations

Research model. 48% of teacher zero-shot at best; the preregistered 60% viability bar is unmet (the data-scaling curve is unbent — see the article). CIFAR absolute numbers are harness-scoped (32px upscale; see the evaluation-path note). Two seeds per objective; one batch size (256); one student size; one consensus family; one deployment frame. The rotation fit consumes 2,500 labeled pairs of the deployment frame.

Related

• —Campaign report: article_cc12m_distillation.md (in this repo)
• —Teacher banks: bulk-cc12m-features · bulk-coco-features
• —Memory-bank line: geolip-vit-captionbank-coco
• —Objective genealogy: loss-manifest

References

• —Radford et al., CLIP — https://arxiv.org/abs/2103.00020
• —van den Oord et al., InfoNCE — https://arxiv.org/abs/1807.03748
• —Yang et al., CLIP-KD — https://arxiv.org/abs/2307.12732
• —Wu et al., TinyCLIP — https://arxiv.org/abs/2309.12314
• —Zhai et al., SigLIP — https://arxiv.org/abs/2303.15343
• —Gower, Generalized Procrustes Analysis — https://doi.org/10.1007/BF02291478
• —Changpinyo et al., Conceptual 12M — https://arxiv.org/abs/2102.08981
• —Cherti et al., OpenCLIP scaling — https://arxiv.org/abs/2212.07143