rwightman/naflex_ViT-B-32.cc12m-pj24-32-40

Model card for naflex_ViT-B-32.cc12m-pj24-32-40

Two-tower CLIP with a NaFlexVit B/32 vision tower trained with patch-size jitter: each batch is patchified at 24, 32 or 40 px (uniform), the base 32 px projection weight resampled on the fly (FlexiViT-style pseudo-inverse resize, timm enable_patch_interpolator), on top of the usual NaFlex sequence-length sampling. Otherwise identical to naflex_ViT-B-32.cc12m, the fixed patch-32 run this card compares against. Loads with naflex_patch_interp enabled, so it accepts any patch size at inference; see the sweep below for which ones are worth using.

Research checkpoint from a NaFlex patch-size jitter experiment on CC12M (variable patch size + variable sequence length training); not a production model.

Model Details

• —Dataset: pixparse/cc12m-wds
• —Epochs: 32
• —Precision: bf16 amp
• —Weight decay: 0.25
• —Objective: InfoNCE
• —Vision tower: timm NaFlexVit base, base patch 32, naflex_patch_interp=true
• —Patch sizes (train): 24 / 32 / 40 px, sampled uniformly per batch
• —Sequence lengths (train): 36 / 49 / 64 / 81 tokens, 200704 tokens per local batch
• —Text tower: classic CLIP transformer, BPE vocab 49408, ctx 77
• —Optimizer: nadamuon, lr 3e-3, warmup 10k
• —Batch size: 4096

Model Usage

import torch
import open_clip

model, preprocess = open_clip.create_model_from_pretrained('hf-hub:rwightman/naflex_ViT-B-32.cc12m-pj24-32-40')
tokenizer = open_clip.get_tokenizer('hf-hub:rwightman/naflex_ViT-B-32.cc12m-pj24-32-40')

image = preprocess(pil_image).unsqueeze(0)
text = tokenizer(['a dog', 'a cat', 'a bird'])
with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)

Model Comparison

Zero-shot ImageNet-1k top-1 (%) as a function of the eval patch size and sequence length, for this model and for naflex_ViT-B-32.cc12m (same config, recipe and data, trained at patch 32 only). Both are epoch-32 checkpoints evaluated with the same harness: OpenAI 80-prompt ensemble, the NaFlex eval transform (each image resized so its patch grid fits the token budget, so a cell's effective resolution is about sqrt(seq_len) x patch; e.g. patch 32 x 49 tokens = 224 px), and the timm patch-embed interpolator enabled for both models. Rows other than 32 for the fixed model are therefore pure post-hoc weight resampling of a patch-32-only model; the delta table isolates what the jitter training adds. Both models trained at 36-81 tokens; 100-256 is extrapolation for both.

This model (patch jitter 24 / 32 / 40)

Fixed patch 32 (naflex_ViT-B-32.cc12m)

Delta: jitter minus fixed (top-1 points)

Reads:

• —Patch size barely matters between 24 and 48 for this model: at any sequence length the four rows sit within ~0.3 pt of each other, including 48 px, which it never trained on. Accuracy is set by the token count. The fixed model needs 144+ tokens before its off-base rows come within a point of base.
• —Jitter is ~free at the base patch (-0.2 to -0.5 pt vs fixed at patch 32, inside the <=0.6 pt repeat-run noise of these runs) and buys 1-2 pt in the cheap regime: at 36-81 tokens every off-base patch is 1.0-2.1 pt better than post-hoc resampling alone (e.g. patch 48 x 36 tokens, a 288 px image in 36 tokens: 38.0 vs 35.8).
• —Post-hoc resampling already covers the high-token regime: at 144+ tokens the fixed model is within a point of its base at every patch >= 24, and at 196-256 tokens it is ~0.5 pt ahead of the jittered model, which decays slightly faster past the trained sequence lengths.
• —Patch 16 does not work for either model (~30 vs ~41): downsampling the 32 px kernel loses information that upsampling to 48 does not, and 16 was outside the jitter range.
• —Sequence length extrapolates for both: both peak at 100-144 tokens against a 36-81 training range (41.4 / 41.7 top-1 at patch 32 x 100 tokens), falling off gently by 256.

The broader benchmark table for the CC12M B/32 comparison runs (ImageNet variants, COCO retrieval, captioning) is on the naflex_ViT-B-32.cc12m card; this checkpoint was not run through that harness.

To evaluate or train at another patch size with the open_clip trainer, pass e.g. --use-naflex --naflex-patch-sizes 24 --naflex-seq-lens 100 (the interpolator is already enabled by this model's config; for other NaFlex checkpoints add --force-naflex-patch-interp).