sona-forge/clip-vit-h-14-image-fp16

Sona Forge — CLIP ViT-H/14 image encoder (ONNX FP16)

Vision-tower-only export of OpenCLIP ViT-H/14 used by IP-Adapter to produce image embeds for SD 1.5 conditioning. Used by the Sona Forge Android app. Pair with `sona-forge/sd15-ipadapter-fp16`.

ONNX shape

How it was made

Pinned conversion environment:

Conversion sequence:

1. 1.Load transformers.CLIPVisionModelWithProjection.from_pretrained("h94/IP-Adapter", subfolder="models/image_encoder", torch_dtype=torch.float16). Note: this loads the OpenCLIP ViT-H/14 weights bundled inside the IP-Adapter repo as a convenience copy.
2. 2.model.eval().
3. 3.torch.onnx.export(model, dummy_pixel_values, opset_version=17, input_names=["pixel_values"], output_names=["image_embeds"], dynamic_axes={"pixel_values": {0: "batch"}, "image_embeds": {0: "batch"}}).

The text branch is not exported. Phase 6 spike characterisation used a deterministic synthetic 512×512 fixture; CLIP image-embeds norm = 21.9 (in the typical 15–25 range for natural portraits).

Files

Licence

OpenCLIP ViT-H/14 — MIT. Original training data is LAION-2B.

Preprocessing

The Sona Forge Android-side ClipImageEncoder uses NEAREST interpolation in the resize step, not BILINEAR. Reasoning: Pillow ≥ 10.0's Image.BILINEAR is anti-aliased by default and produces a small but consistent cosine drift (~0.04) versus a Kotlin pure-bilinear implementation. NEAREST on both sides clears the 0.999 cosine threshold the in-tree JVM golden test asserts. If you re-use this ONNX with a different preprocessor, verify cosine match against your own reference embedding.

Memory footprint

~1.2 GB FP16 on disk; ORT CPU EP promotes to FP32 at session load (~2.4 GB resident). On Android (NNAPI / XNNPack), FP16 runs natively.

Usage

import onnxruntime as ort
import numpy as np
from PIL import Image

clip = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

CLIP_MEAN = np.array([0.48145466, 0.4578275, 0.40821073], dtype=np.float32)
CLIP_STD = np.array([0.26862954, 0.26130258, 0.27577711], dtype=np.float32)

img = Image.open("portrait.png").convert("RGB")
w, h = img.size
side = min(w, h)
img = img.crop(((w - side) // 2, (h - side) // 2, (w + side) // 2, (h + side) // 2))
img = img.resize((224, 224), Image.NEAREST)
arr = (np.asarray(img, dtype=np.float32) / 255.0 - CLIP_MEAN) / CLIP_STD
arr = arr.transpose(2, 0, 1)[None, ...].astype(np.float16)  # NCHW FP16

image_embeds = clip.run(["image_embeds"], {"pixel_values": arr})[0]  # (1, 1024)

Provenance

• —Original weights: `laion/CLIP-ViT-H-14-laion2B-s32B-b79K`.
• —Bundled-copy used by this conversion: `h94/IP-Adapter/models/image_encoder`.