mlboydaisuke/Granite-Embedding-97M-Multilingual-R2-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta, 2026-06).

<!-- gen-cards:devicemark begin (managed by scripts/gen-cards + tools/devicemark_row.py — edit cards.json, not this block) --> This model has no row on DeviceMark, the on-device LLM leaderboard. <!-- gen-cards:devicemark end -->

Granite-Embedding-97M-Multilingual-R2 — Core AI export

Zoo card, recipe and gate transcript: coreai-model-zoo/models/granite-embedding-97m.

IBM's 97M-parameter multilingual text embedder — a ModernBERT encoder, 384-d CLS-pooled unit vectors, Japanese and English among its languages — as a static .aimodel for macOS 27 and, ahead-of-time compiled, for the iPhone 17 Pro. `ibm-granite/granite-embedding-97m-multilingual-r2` (Apache-2.0, revision 835ad1408…) is the smallest embedder in this catalog (390 MB fp32, against 1.2 GB for EmbeddingGemma-300m and 1.1 GB for Qwen3-Embedding-0.6B) and its first encoder-architecture one — every other embedder here is a causal decoder run as an encoder. Its retrieval quality relative to those three was not measured here: the fixture set below is a parity instrument (35 texts, 4 queries, 12 documents), not a benchmark.

This is an encoder, not a generator — one forward over the right-padded grid returns one unit vector. No autoregressive loop, no KV cache, no LM head. It runs as a plain .aimodel through raw AIModel.run (like the vision encoders), not the pipelined generate engine.

Architecture (model_type: modernbert): 12 layers, hidden 384, 12 heads × 32, GLU MLP 1536 (SiLU), vocabulary 180,000, biasless everything (attention, MLP, LayerNorm ε 1e-5). Global attention at layers 0, 3, 6, 9 (RoPE θ 150,000); the other eight are local, a sliding window of inclusive radius 64 (129 keys per interior query, RoPE θ 160,000). Layer 0 has no attention pre-norm (the embedding LayerNorm serves). Pooling is CLS → L2 normalize, both in the graph.

Graph contract

input  "input_ids"       [1, S]    int32   right-padded to the grid S with 179935
input  "attention_mask"  [1, S]    int32   1 over real tokens, 0 over padding
output "embedding"       [1, 384]  fp32    CLS-pooled, L2-normalized
S = 128 or 512 (export-time choice); batch = 1

Host recipe — the tokenizer is the whole contract, and the stock one is not enough:

• —No prefix, no stripping, no normalization. Query and document prompts are both empty in the checkpoint. Raw whitespace is kept: sentence-transformers strips text before tokenizing, the upstream README's AutoTokenizer path does not, and the two disagree on " 東京駅から…\n". The reference is the raw path.
• —Tokenize with the pinned tokenizer.json: regex Split(Isolated) → ByteLevel (no prefix space) → byte BPE with `ignore_merges = true` (a whole pre-token that is in the vocabulary wins; ક is token 2999, not three). A BPE that ignores the flag tokenizes differently.
• —Truncate the body to S−2, then wrap: [CLS 179934] body… [SEP 179938], right-pad with PAD 179935 and mask 0. Truncating after adding the specials loses SEP; padding with 0 is a different token. Both are silent.
• —Similarity = dot product (unit vectors). Dimension truncation is not a property of this model.

`conversion/granite_embedding/_granite_tokenizer.py` is that recipe with no HF import, and host/GraniteTokenizer.swift in this repo the same recipe in Foundation-only Swift; the gate holds both to AutoTokenizer exactly (ids and masks) over 681 texts × 2 grids = 1,362 cases including every added token in five boundary contexts, and proves four mutations are caught (pad 0 / lose SEP / strip / ignore_merges=false).

Measured

iPhone 17 Pro (iPhone18,1), iOS 27.0 build 24A437, the compiled h18p bundles loaded by the native AIModel loader, GPU-preferred (MPSGraph/Metal plan). Every row: 35 HF texts, the gate below, 105 warm samples, thermal state fair before and after, caches retained (so "first" is process-first, not cache-cold). Peak footprint is the whole app process, tokenizer and file hashing included. Measured 2026-09-19.

Each row matched 4/4 retrieval top-1s with 0 clear-pair flips and 0 repeat drift.

Mac (M4 Max, Mac16,9), macOS 27.0 build 26A428, the JIT .aimodel, GPU-preferred, fp32. The driver refused to run while any foreign accelerator job was present; 105 warm samples.

The Mac h16c AOT twin also passed 70/70 (same numerics), but its timings were taken with another lane's GPU evaluation running and are not reported. The w8 variant on Mac is gated on CPU only (min cosine 0.999410, max |err| 5.67e-3, ranking exact); Mac GPU for w8 was not run.

The fixed grid computes every position, so pick the smallest grid that covers the text: S=128 for queries and short notes, S=512 for passages. fp32 is the default. w8 is a storage option only — 22% smaller, not faster here — because the 180,000×384 fp32 vocabulary table is 276 MB of the bundle and palettization touches the 48 linear weights alone.

Numerics gate

One gate at every stage, the oracle being official HF eager CPU fp32 (transformers 4.57.6): per text cosine ≥ 0.999, max element error ≤ 0.02, L2-norm error ≤ 0.002; per query exact top-1 over the 12 documents, retrieval-score error ≤ 0.01, and no inversion of any document pair the oracle separates by ≥ 0.001; repeat drift ≤ 1e-6. A wrong-pairing control (every vector matched to the wrong text) must FAIL.

• —Authoring (gate_granite_authoring.py): the re-authored graph against every one of the 13 saved hidden states, max |err| ≤ 1e-4 at fp32, both grids. Five mutations must trip it: all-global, all-local, ignore-padding and mean-pooling fail the embedding gate; a local radius of 63 instead of 64 passes the embedding gate (cos 0.99995) and fails only the layer gate — which is why the layer gate exists. Whole-model fp16 fails this layer gate on both grids.
• —Export: the torch-exported, decomposed graph is gated before conversion, on both grids.
• —Runtime: Mac CPU and GPU (JIT), Mac h16c AOT, iPhone h18p AOT — the tables above.
• —w8: the same gate at prepared, finalized and decomposed stages, 48 lut_to_dense ops counted, palettes hashed; the iOS w8 export reuses the Mac palettes byte for byte.

`models/granite-embedding-97m/gate-granite-embedding-97m.json` in the zoo is the transcript: the eight runtime rows (min cosine, max error, retrieval, timings, device/OS build), the tokenizer gate and the authoring gate, each with the sha256 of the full record it summarizes.

⬇️ Bundle

This repo — one folder per variant, each self-contained: the bundle, tokenizer/, reference.json (the 35 HF fixtures with ids, masks and embeddings — the parity test) and provenance/ (export manifest with per-file sha256, the runtime gate record). coreai-kit.json at the root maps platform → folder.

The ios/ bundles are compiled for one device architecture (h18p, the iPhone 17 Pro) with xcrun coreai-build compile --platform iOS --min-deployment-version 27.0 --preferred-compute gpu --architecture h18p (coreai-build 3600.83.1). Never load an iOS bundle on a Mac. Other phones need their own compile from the recipe; the source IR is reproducible, not shipped.

Convert yourself: `conversion/granite_embedding/` — five staged scripts; `recipe.toml` names the commands.

CoreAIKit (Swift)

Not enrolled in the kit catalog. The kit's TextEmbedder pads with 0, truncates after adding the special tokens (losing SEP), applies its own BPE without ignore_merges, discovers a single .aimodel, and has no grid / architecture selection — every one of those is wrong for this model. Running it today means: the Swift tokenizer from this repo's host/ folder, a fixed grid, AIModel on the platform's folder. Enrolling it needs a textEmbedding driver that takes the pad id, a SEP-preserving truncation, a per-platform variant path and an AOT-aware loader — tracked as maintainer work, not a blocker on the bundle.

The port in one lesson: gate the layers, not just the vector

ModernBERT's alternating local/global attention is the whole risk. The config says local_attention: 128; the executed window is inclusive |i − j| ≤ 64 — 129 keys — and a window of 63 reproduces the final embedding to cos 0.99995 while every hidden state past layer 1 is wrong. Only a per-layer oracle catches it. Three more things the raw checkpoint settles that the modeling file hides: layer 0 has no attention norm (adding one loads a missing weight), the two RoPE thetas are per-layer-kind, and the CLS/L2 head needs an explicit clamp_min epsilon because the converter's F.normalize decomposition drops it.

License and limits

Apache-2.0 at the pinned upstream revision; this repo carries IBM's unmodified card as UPSTREAM_README.md and a LICENSE-NOTE.md listing the changes (static graph, in-graph pooling, optional w8 palettes, h18p compile). Not tested: other phones or OS builds, the Mac GPU with w8, the Neural Engine, dynamic or batched shapes, S > 512, languages beyond the JA/EN fixtures, retrieval quality on a benchmark, sustained thermals, true cache-cold load.