nodevorg/siglip2-base-patch16-256-coreml

1---2license: apache-2.03base_model: google/siglip2-base-patch16-2564pipeline_tag: feature-extraction5tags:6  - coreml7  - siglip28  - siglip9  - clip10  - image-text11  - embeddings12  - vision13  - apple-silicon14---15 16# SigLIP2 base-patch16-256 — Core ML17 18Core ML (`.mlpackage`) conversion of [`google/siglip2-base-patch16-256`](https://huggingface.co/google/siglip2-base-patch16-256) for on-device text and image embedding on Apple platforms. The image and text towers ship as separate models; both produce 768-dimensional, L2-normalized embeddings in a shared space, so image↔text similarity is a single dot product.19 20Converted and maintained by [nodevorg](https://huggingface.co/nodevorg) for [Alarian](https://alarian.ai)'s semantic frame search. Not affiliated with Google.21 22## Files23 24| file | size | sha256 |25|---|---|---|26| `ImageEncoder.mlpackage.zip` | 163 MiB | `406938456c8f8e91f01632cb74131fdfb993064d8d0bc8f6fbbd142709f1ba66` |27| `TextEncoder.int8emb.mlpackage.zip` — **recommended** | 323 MiB | `11471872101a6ca88dd51ed9de668b728ae9d09c0f853aeac8a6cc96f1c109d6` |28| `TextEncoder.mlpackage.zip` | 506 MiB | `4f31f38a1729a0044f0ffc95a940dc1a6a2fdcd5fcf78a03cbb662bada6cbafb` |29| `tokenizer.json` | 33 MiB | `cb9140fae3ac5122c972d37adf83e1248471a38147ad76f8215c8872c6fd8322` |30| `tokenizer_config.json` | — | `14afe629fe4959b9e0d51e1852b8d9f7ad074f90a1a7125a4fcdd17f06e78fc8` |31| `special_tokens_map.json` | — | `baec30ea10906f16adb8c18af7a34023002c1746542612b8b41c9f09e1351351` |32 33`TextEncoder.int8emb` is a drop-in replacement for the fp16 `TextEncoder` — identical input/output names, shapes, and dtypes — with the token-embedding table quantized to int8 (36% smaller download, worst-case parity 0.9996; details below). Use the full-fp16 text encoder only if you need maximum precision.34 35The tokenizer files are copied verbatim from the source checkpoint (Gemma BPE, 256k vocabulary).36 37## Model interface38 39### ImageEncoder40 41- **Input** — `pixel_values`: Core ML `ImageType`, RGB, 256×256. Preprocessing is baked into the model (`scale=1/127.5`, `bias=[-1,-1,-1]`, matching the checkpoint's `SiglipImageProcessor` configuration): resize to 256×256 and pass raw `[0, 255]` RGB — no caller-side normalization. Resize with an aspect-ratio squash rather than a center crop; that matches training.42- **Output** — `image_embedding`: 768-dim float16, L2-normalized.43 44### TextEncoder (both variants)45 46- **Input** — `input_ids`: 1×64 int32 token ids. Tokenize with the included tokenizer using `padding="max_length"`, `max_length=64`, lowercased input (`do_lower_case=true`). There is deliberately **no `attention_mask` input**: SigLIP's text tower trains with unmasked bidirectional attention over the fixed 64-token sequence and pools the last sequence position, so a mask would deviate from training.47- **Output** — `text_embedding`: 768-dim float16, L2-normalized.48 49Because both outputs are normalized in-graph, cosine similarity is `dot(image_embedding, text_embedding)`.50 51## Usage (Swift)52 53```swift54import CoreML55 56let image = try MLModel(contentsOf: compiledImageEncoderURL)57let text  = try MLModel(contentsOf: compiledTextEncoderURL)58 59// Image: 256×256 RGB CVPixelBuffer, raw [0, 255] values60let imageOut = try image.prediction(61    from: MLDictionaryFeatureProvider(dictionary: ["pixel_values": pixelBuffer]))62let imageEmbedding = imageOut.featureValue(for: "image_embedding")!.multiArrayValue!63 64// Text: 64 int32 token ids (e.g. via swift-transformers' Tokenizers module,65// loading the tokenizer.json from this repo), lowercased, padded to length 6466let ids = try MLMultiArray(shape: [1, 64], dataType: .int32) // fill with token ids67let textOut = try text.prediction(68    from: MLDictionaryFeatureProvider(dictionary: ["input_ids": ids]))69let textEmbedding = textOut.featureValue(for: "text_embedding")!.multiArrayValue!70 71// Similarity = dot product (embeddings are already L2-normalized)72```73 74## Architecture note75 76The source checkpoint's `config.json` declares `model_type: siglip`, not `siglip2`. At this fixed-resolution, non-NaFlex tier, the SigLIP2-trained weights are architecturally identical to SigLIP v1 — `transformers.AutoModel` resolves the checkpoint to `SiglipModel`, and that is what was converted. Relative to SigLIP v1, only the weights and the tokenizer (Gemma BPE 256k, vs. 32k SentencePiece) differ.77 78Most of the text encoder's size is the 256k-row token-embedding table (256000 × 768 × 2 bytes ≈ 375 MiB at fp16), which is why quantizing that single tensor recovers most of the possible size savings.79 80## Quantized text encoder81 82`TextEncoder.int8emb.mlpackage` is produced by post-training quantization of the converted fp16 model via `coremltools.optimize.coreml`, applying `OpLinearQuantizerConfig(mode="linear_symmetric", dtype="int8", granularity="per_channel")` to **only** the `text_model.embeddings.token_embedding` weight (op-name-targeted); attention and MLP weights remain fp16. The embedding table is a gather weight — each forward pass reads one row per token, so quantization error does not compound through a matmul chain.83 84A whole-model int8 variant was also evaluated: ≈270 MiB unzipped (vs. ≈352 MiB for embedding-only) but with worst-case parity of 0.9964 vs. 0.9996. The embedding-only variant was published as the better size/precision trade-off for retrieval workloads, where score margins matter.85 86## Parity vs. PyTorch reference87 88Cosine similarity between the PyTorch fp32 reference (`transformers`) and the converted Core ML models, on a fixture of 4 images (solid red/green/blue, synthetic sunset gradient) × 4 texts. Acceptance gate: > 0.99 per item, with unchanged text↔image rankings.89 90### fp16 encoders91 92| item | cosine |93|---|---|94| image: solid_red | 0.9999988 |95| image: solid_green | 0.9999992 |96| image: solid_blue | 0.9999992 |97| image: gradient_sky | 0.9999987 |98| text: "a red square" | 0.9999995 |99| text: "a photo of a sunset over the ocean" | 0.9999997 |100| text: "a solid blue color" | 0.9999995 |101| text: "a green field" | 0.9999996 |102 103Worst case **0.9999987**. Text↔image rankings match the PyTorch reference exactly across all 4×4 pairs.104 105### int8-embedding text encoder106 107| text | cosine |108|---|---|109| "a red square" | 0.99978 |110| "a photo of a sunset over the ocean" | 0.99976 |111| "a solid blue color" | 0.99962 |112| "a green field" | 0.99975 |113 114Worst case **0.99962**; rankings identical to both the PyTorch reference and the fp16 model.115 116## Conversion details117 118Converted from the Hugging Face `transformers` reference implementation. Each tower was traced independently (`torch.jit.trace` on thin wrappers around `SiglipModel.vision_model` / `.text_model`) and converted with:119 120- `convert_to="mlprogram"`, `compute_precision=FLOAT16`121- `minimum_deployment_target`: macOS 15122 123Pinned toolchain: `torch==2.7.0`, `transformers==5.14.1`, `coremltools==9.0`, `numpy==2.3.5`. Note that `numpy>=2.4.0` breaks the `coremltools==9.0` PyTorch frontend for models using `torch.nn.MultiheadAttention` (SigLIP's vision pooling head) — see [apple/coremltools#2633](https://github.com/apple/coremltools/issues/2633); pin `numpy<2.4.0` until the fix ships in a release.124 125## License126 127Apache-2.0, inherited from the base model. Free for commercial and closed-source use.128