Liangzhi-Li/clipbench-blending

CLIPBench-Blending

Code, cached features and complete per-cell records for The Blending Ratio Is Not Where the Performance Is: Diagnosing Prototype Blending for Few-Shot Adaptation of Vision–Language Models (Liangzhi Li, Bowen Wang, Yiming Qian, Thorsten Neumann, Xia Xie, and Guangshun Li; corresponding author Guangshun Li).

A large family of few-shot adaptation methods for vision–language models classifies with a convex combination of the zero-shot text prototype and the mean of the K labelled image features, governed by a blending ratio that is routinely tuned on held-out labels. This repository contains the study that asks what that ratio is worth: whether it can be computed rather than searched, whether it can be estimated without validation data, and whether setting it perfectly would make the resulting classifier competitive.

Headline numbers, all paired within cells and aggregated by a cluster bootstrap over the ten datasets:

Artefacts

Cached features, text prototypes and the full record set are released separately, because the point of this benchmark is that nobody should have to re-encode a single image to check it:

• —Hugging Face: <https://huggingface.co/datasets/Liangzhi-Li/clipbench-blending> (35 GB: image and text features for every cell, the CoOp-split re-extraction, all 50 augmented-view caches, and the full record set)

A DOI-bearing archival copy is reserved and will be linked here once the preprint is posted.

See `DATA.md` for what lives where and how to verify a download. `release/COVERAGE.md`, generated from the repository itself, confirms that every cell the records report has a published feature cache behind it.

Layout

Reproducing the paper from the released records

No GPU, no images. Download the record set, then:

pip install numpy scipy matplotlib torch
bash scripts/build.sh          # regenerates every table and figure, compiles the PDF
python3 scripts/consistency_scan.py   # asserts the PDF agrees with the records

scripts/build.sh runs code/paper_assets.py, code/paper_assets_ext.py, code/probe_tables.py, code/capacity_macros.py and the two figure scripts in order, then scripts/fit_tables.py (which must run last: the generators emit plain tabulars, and four of them are wider than a column), then three pdflatex passes for the main paper and the same for paper/supplementary.tex.

Reproducing the experiments from the released features

Point CLIPBENCH_FEATS at an unpacked feats/ directory, or pass --feat:

# main matrix: 10 datasets x 5 backbones x 5 shot counts x 5 seeds x 4 prompt tiers
python3 code/run_formal.py --feat feats --datasets dtd --backbones ViT-B-16 \
    --out results/mine.json

# augmentation arm (needs the supaug view caches; see DATA.md)
python3 code/run_aug.py --feat feats --datasets dtd --backbones ViT-B-16 \
    --out results/mine_aug.json

# CoOp-split arm
python3 code/run_formal.py --feat feats_coop --prompts ds --datasets dtd \
    --backbones ViT-B-16 --out results/mine_coop.json

# the two geometry probes and the offset counterfactual
python3 code/probe_oracle_geometry.py --feat feats --out results/mine_probe.json
python3 code/delta_counterfactual.py  --feat feats --out results/mine_dc.json

# capacity theorem + anisotropic sampling law, measured per stratum (Secs. 3.3, 3.5)
python3 code/probe_capacity.py --feat feats --out results/mine_capacity.json

# modern-encoder arm (Sec. 6.8): SigLIP2 runs the standard matrix; the
# text-free encoders have no blending ratio and run as probe references.
# Their feature caches regenerate with code/extract_gpu.py --backbones
# SigLIP2-B-16 DINOv2-B-14 DINOv3-B-16 (one decode pass covers all three).
python3 code/run_formal.py --feat feats --datasets dtd --backbones SigLIP2-B-16 \
    --prompts ds --out results/mine_siglip2.json
python3 code/run_vision_only.py --feat feats --datasets dtd \
    --vision-backbone DINOv3-B-16 --out results/mine_vision.json

Shards are independent; scripts/pool.sh runs a queue of <dataset> <backbone> pairs across N workers, and code/merge_shards.py merges them while asserting that no shard is missing and no cell was written twice.

Reproducing the features from images

python3 code/extract_gpu.py  --datasets dtd eurosat --out feats --scratch scratch
python3 code/extract_aug.py  --datasets dtd eurosat --out feats --scratch scratch
python3 code/extract_coop.py --datasets dtd --zips <coop-zips> --root <images> \
    --out feats_coop

Extraction streams the pinned clip-benchmark webdataset shards, encodes them with all five backbones in one pass, and deletes each shard once its features are written; peak disk stays a few GB even for SUN397. It is resumable: re-run the same command after any interruption.

The CuPL prompt tier needs an OpenAI-compatible endpoint:

export LLM_BASE_URL=https://<endpoint>/v1/
export LLM_API_KEY=...
python3 code/gen_descriptions_wds.py --budget-usd 2.00
python3 code/encode_descriptions_wds.py --feat feats

Verification built into the pipeline

• —code/test_shrinkage.py — 27 unit tests on the estimators.
• —code/test_baselines.py — baseline reimplementations against synthetic data with known answers, plus a smoke test on the released feature cache.
• —code/synth_validate.py — unbiasedness of ĝ² against known synthetic truth.
• —code/verify_theory_ext.py — Monte-Carlo checks of the closed forms.
• —code/verify_capacity.py — Monte-Carlo checks of the capacity theorem, the Baranchik dominance certificate, and the anisotropic sampling law (V1–V5), run before any of them entered the paper.
• —code/probe_capacity.py — measures all three on the released features; code/capacity_macros.py turns the output into the paper's macros and the supplementary per-stratum table.
• —code/check_dinov3.py — asserts a (possibly mirrored) DINOv3 checkpoint is the published architecture, with the declared preprocessing and sane features, before any experiment may depend on it.
• —code/modern_macros.py — turns the modern-encoder records into the paper's macros and the supplementary per-K tables.
• —code/sanity_zeroshot.py — zero-shot accuracy must land inside a band of published CLIP numbers before a feature cache is trusted.
• —code/check_device_equiv.py — CPU/GPU solver equivalence.
• —code/boot_coverage.py — coverage simulation for the cluster bootstrap.
• —code/precision_check.py — float64 vs float32 solver sensitivity.
• —scripts/consistency_scan.py — every number quoted in the PDF is traced back to the records that produced it.

Conventions

• —POOL supplies support sets and the population statistics behind the MSE-oracle; TEST is touched only by the final evaluation and by quantities explicitly named *_oracle, which are reported as unreachable bounds.
• —Cells whose per-class pool residue falls below five after support sampling are dropped and counted, never silently skipped.
• —Features are ℓ2-normalised once at extraction; blended prototypes are re-normalised before cosine scoring.
• —All accuracy comparisons are paired within cells; cross-cell aggregates resample datasets, not cells, with a Student-t correction for ten clusters.

Licence

MIT for the code and derived artefacts. Dataset images belong to their original providers under their original licences; only derived features are distributed.