litert-community/LFM2.5-Encoder-350M

LFM2.5-Encoder-350M — LiteRT

LiquidAI/LFM2.5-Encoder-350M converted to LiteRT (.tflite) for on-device inference. A multilingual (15 languages) bidirectional encoder on the LFM2 hybrid backbone — gated short-convolutions plus grouped-query attention — for embeddings, retrieval, classification heads, and masked-token prediction, fully offline on CPU.

Model description

All signatures take batch-1, right-padded static shapes: input_ids int32 [1, S] and attention_mask int32 [1, S] (1 = real token, 0 = pad).

Padded positions are fully masked inside the graph, in both the convolution path and attention, so the output at valid positions does not depend on how much padding follows: encode_64, encode_128 and encode_256 agree bitwise on the same sentence and match the unpadded PyTorch reference.

For a smaller sibling see LFM2.5-Encoder-230M.

How to use

1. Install dependencies

pip install ai-edge-litert numpy tokenizers huggingface_hub

2. Save the script below as embed.py:

#!/usr/bin/env python3
"""Embed sentences with litert-community/LFM2.5-Encoder-350M and rank them by similarity."""
import argparse

import numpy as np
from ai_edge_litert.interpreter import Interpreter
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

REPO = "litert-community/LFM2.5-Encoder-350M"
MODEL_FILE = "LFM2.5-Encoder-350M_wi8fc.tflite"


def embed(runner, tokenizer, text, seq_len):
    """Mean-pools the encoder states over the real tokens into one vector."""
    ids = tokenizer.encode(text).ids
    if len(ids) > seq_len:
        raise SystemExit(f"{len(ids)} tokens exceed --seq-len {seq_len}")
    input_ids = np.zeros((1, seq_len), np.int32)
    attention_mask = np.zeros((1, seq_len), np.int32)
    input_ids[0, : len(ids)] = ids
    attention_mask[0, : len(ids)] = 1
    states = list(runner(input_ids=input_ids, attention_mask=attention_mask).values())[0]
    vector = states[0, : len(ids)].mean(axis=0)
    return vector / np.linalg.norm(vector)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--query", required=True, help="The sentence to match.")
    parser.add_argument("--candidate", action="append", required=True,
                        help="A candidate sentence, repeatable.")
    parser.add_argument("--seq-len", type=int, default=128,
                        choices=[64, 128, 256, 512])
    parser.add_argument("--threads", type=int, default=8)
    args = parser.parse_args()

    model_path = hf_hub_download(REPO, MODEL_FILE)
    tokenizer = Tokenizer.from_file(hf_hub_download(REPO, "tokenizer.json"))
    interpreter = Interpreter(model_path=model_path, num_threads=args.threads)
    runner = interpreter.get_signature_runner(f"encode_{args.seq_len}")

    query = embed(runner, tokenizer, args.query, args.seq_len)
    scored = [(float(query @ embed(runner, tokenizer, c, args.seq_len)), c)
              for c in args.candidate]
    for score, text in sorted(scored, reverse=True):
        print(f"{score:6.3f}  {text}")


if __name__ == "__main__":
    main()

3. Run it

python embed.py --query "What is your refund policy?" \
  --candidate "Our refund policy allows 30 days" \
  --candidate "Steps to recover a forgotten login" \
  --candidate "The weather in Osaka is mild in spring"

 0.604  Our refund policy allows 30 days
 0.579  The weather in Osaka is mild in spring
 0.570  Steps to recover a forgotten login

Mean-pooling the raw encoder states is the simplest sentence representation and is what the numbers above use; for retrieval at quality you would normally train a pooling head or fine-tune on your own pairs. For masked-token prediction use the mlm_128 signature and read the logits at the [MASK] position.

On Android/iOS use the LiteRT runtime's SignatureRunner APIs with the same signature names; the tokenizer is the standard Hugging Face tokenizer.json, which the Rust/Swift/Kotlin tokenizers bindings all read.

Performance

int8 (wi8fc) file, CPU only.

Mac figures are the median of 20 warm runs (ai-edge-litert 2.1.6, XNNPACK, otherwise idle machine). iPhone figures come from the on-device gate (TFLite C API + SignatureRunner + XNNPACK); each is the last of three consecutive measurements, not a median.

Budget for one slow first call. The first inference after loading pays a one-time graph preparation. On the Mac that first call took 908 ms against a 36.5 ms steady state; on the iPhone the three consecutive encode_128 measurements were 102, 51 and 42 ms, and the three encode_512 measurements were 229, 152 and 145 ms. Model load was 1.48 s on the iPhone, with a peak footprint of about 1.65 GiB.

Those three consecutive values are not a language effect — the signatures are fixed-shape, so every input of a given signature costs the same. Measured warm on the Mac with the run order reversed, one signature takes 36.4 / 37.3 / 36.6 ms on English, Japanese and Arabic sentences of 17, 21 and 27 tokens.

Thread count matters more than anything else here: at the interpreter default this model measures 52.4 ms and 227.1 ms for encode_128 and encode_512, against 36.5 ms and 111.2 ms at 8 threads.

Accuracy note

Parity against the PyTorch fp32 reference over 16 sentences covering all 15 supported languages — mean-pooled sentence-embedding cosine against the original Lfm2BidirectionalModel, plus top-5 fill-mask agreement on English, French, German and Japanese cloze prompts:

On the iPhone 17 Pro the int8 file reproduces the Mac outputs bit-exactly — cosine 1.000000, max absolute difference 0.0 — across every tested language and signature.

Android (Pixel 8a)

Android figures use the standard TFLite `benchmark_model` on a Pixel 8a (Tensor G3, Android 16) — 5 warm-up runs then 20 timed runs, the signature selected explicitly with --signature_to_run_for, CPU at 4 threads.

GPU works as of the 2026-08-13 re-export — via the single-signature file. This re-export respells the one idiom mobile GPU delegates refuse — transformers' rank-5 repeat_kv expand — into an equivalent rank-4 matmul (outputs bitwise-identical on CPU), which makes the graph fully GPU-delegable. The 5-signature file does not compile on the Pixel 8a OpenCL or iPhone 17 Pro Metal delegates: each signature subgraph materializes its own fp32 weight copy (~7 GB total for this model), which exceeds those phones' memory. It does compile and run on the Adreno GPU of a Galaxy S26 under LiteRT CompiledModel 2.2.0 — 82.04 ms — but it pays a 17.3 s load there against 4.07 s for the single-signature file (see Snapdragon NPU (Hexagon) below). The new `LFM2.5-Encoder-350M_wi8fc_single-sig.tflite` (encode_512 only, 359 MB) is the phone-GPU artifact: Pixel 8a OpenCL full delegation, encode_512 12.2 ms, cosine 0.9994 vs the fp32 desktop reference (LiteRT CompiledModel API, fp32 GPU precision, best of 3 warm runs); iPhone 17 Pro Metal 171 ms, cosine 0.9994. The 5-signature file remains the CPU artifact (bit-exact on device; iPhone encode_512 144–158 ms). Set the GPU precision to fp32 — at fp16 GPU precision this family's norm reductions overflow and every output is NaN. These CompiledModel timings are not comparable to the classic-delegate benchmark_model timings above (different GPU runtime).

Snapdragon NPU (Hexagon)

• —LFM2.5-Encoder-350M_fp16.tflite — neither accelerator produced a usable row on the S26. Both ended the same way: LiteRtException: Failed to compile model.
• —LFM2.5-Encoder-350M_wi8fc.tflite — the GPU runs it at 82.04 ms. The NPU does not — LiteRtException: Failed to compile model.
• —LFM2.5-Encoder-350M_wi8fc_single-sig.tflite — the GPU runs it at 82.24 ms. The NPU does not — the graph compiles and then fails to run (LiteRtException: Failed to invoke the compiled model).

Measured on a Samsung Galaxy S26 (Snapdragon 8 Elite Gen 5 / SM8850, Hexagon v81, Android 16) with LiteRT CompiledModel 2.2.0, one accelerator per process, 5 warm-up runs then N=50 timed runs, median reported. Every run held thermal status NONE throughout. Headroom 0.72–0.78, where 1.0 is the throttling threshold.

GPU wiring: GPU guide. NPU recipe: NPU guide.

License

LFM Open License v1.0 (see LICENSE, unchanged from the base model). Note the license's commercial-use threshold (Section 5). This repository redistributes converted Derivative Works of LiquidAI/LFM2.5-Encoder-350M with modification notices per Section 4; all credit for the model to Liquid AI.