mlboydaisuke/multilingual-e5-base-ExecuTorch

1---2license: mit3tags:4- executorch5- xnnpack6- pte7- on-device8- sentence-similarity9- feature-extraction10base_model:11- intfloat/multilingual-e5-base12base_model_relation: quantized13---14# multilingual-e5-base — ExecuTorch15 16Multilingual E5, base-sized — one index over 100 languages. Text in, one17768-dimensional vector out, for search and retrieval that never leaves the18device.19 20- **Source**: intfloat/multilingual-e5-base — 12 layers, 768 dimensions, 250,002 vocabulary21- **License**: mit22- **Input**: `input_ids` and `attention_mask`, both `[1, 256]` int6423- **Output**: `[1, 768]`, mean-pooled and L2-normalised inside the graph24 25## The recipe is in the graph, and it was read off this repo26 27sentence-transformers stores it per model, and the shelf's seven embedding models do28not agree. This one pools **mean** and29**normalises**, read from30`1_Pooling/config.json` and `modules.json` rather than inferred from the family name.31Getting it wrong does not throw; it returns vectors that look fine and rank wrong.32 33## The prefix is not in the graph34 35This model is trained with `query: ` in front of the text and expects it at36inference. That happens before tokenisation, so the `.pte` never sees it as anything37but tokens — and leaving it out does not throw. It returns a plausible vector that38retrieves worse.39 40## Verification41 42| build | file | size (MB) | Mac ms* | backend takes | worst cosine vs eager | retrieval budget |43|---|---|---|---|---|---|---|44| fp32 | `embed_multilingual_e5_base_xnnpack_fp32.pte` | 1110.0 | 32.5 | 77.3% | 1.000000 | 0% |45| fp16 | `embed_multilingual_e5_base_xnnpack_fp16.pte` | 555.2 | 53.1 | 66.7% | 0.999999 | 6% |46| Core ML (fp16, iOS) | `embed_multilingual_e5_base_coreml_all.pte` | 555.7 | 6.8 | 100.0% | 0.999994 | 19% |47 48\*Mac arm64, one 256-token sequence, **fastest of five medians of ten** — a reference49point for relative cost, not a device number. The host shares its cores with other work,50and a single median does not survive that: the same eager model here measured 19.6 ms and51182.8 ms twenty minutes apart. Contention only ever adds time, so the fastest repetition is52the one that means something. Torch eager fp32, measured the same way, is5334.7 ms.54 55Cosine is measured against the model run in eager through its own pooling, over eight56sentences. The last column is the one that decides: rank those eight against each57other, and ask whether this build's score error is smaller than the gap between the58document a query retrieves and the runner-up. Every shipped build keeps all eight59top-1 results.60 61## The attention is eager, and that is the faster export62 63`F.scaled_dot_product_attention` does not survive export as one operation. The edge64dialect lowers it through `_safe_softmax`, whose guard against a row with no unmasked key65at all leaves **11 operations XNNPACK cannot take, in every attention66block** — `scalar_tensor`, `where`, `mul.Scalar`, `logical_not`, `eq`, `full_like`, `any.dim`. Each one cuts the subgraph in two.67 68The switch is `attn_implementation="eager"`: transformers then builds the mask69itself, as `torch.finfo(dtype).min`, instead of handing `F.sdpa` a **boolean** mask70for PyTorch to fill with `-inf`.71 72The guard is emitted whether or not it can ever fire, and here it cannot: it triggers only73on `-inf`, and this arm never produces one. So the two differ only about rows that have no74unmasked key at all — sdpa zeroes them, this one gives them a uniform row — and those are75padding rows, which the pooling discards and which every real query row masks out anyway.76Measured with all but eight positions masked, as adversarial as this shape gets, the two77graphs agree to 1.4e-07.78 79XNNPACK fp32 goes from **61.9% to 77.3%** delegated.80 81## Not shipped: int882 83`embed_multilingual_e5_base_xnnpack_int8.pte` is **855.5 MB** against fp16's 555.2 MB. Dynamic int8 quantises the84linear weights and leaves the token embedding table in fp32, and here that table is85768 MB of the 1110.0 MB model — **69%**. The size a build comes out at is86`0.5 + 1.5 x (table share)` times the fp16 build; at 69% that is871.54, so there was never a smaller file to be had.88 89It is withheld on the number that decides. Ranking the eight test sentences against90each other, this build moves a pair score by at most **0.0057** while the91closest fp32 decision — the gap between the document a query retrieves and the92runner-up — is **0.0022**. That is **255%** of93the room available, against a bar of 50%.94 95Correlation reads 0.997142 for this build, which no correlation gate96would stop.97 98torch.export -> to_edge_transform_and_lower(partitioner) -> .pte99(conversion scripts: [executorch-models](https://github.com/john-rocky/executorch-models))100