mlboydaisuke/LFM2.5-1.2B-Instruct-ExecuTorch

1---2license: other3tags:4- executorch5- xnnpack6- pte7- on-device8- text-generation9base_model:10- LiquidAI/LFM2.5-1.2B-Instruct11base_model_relation: quantized12---13 14# LFM2.5-1.2B-Instruct — ExecuTorch XNNPACK 8da4w15 16`lfm2_5_1_2b_xnnpack_8da4w.pte` (741 MB)17 18- **Source**: LiquidAI/LFM2.5-1.2B-Instruct (hybrid conv/attention)19- **License**: LFM Open License v1.020- **Quantization**: 8da4w (8-bit dynamic activation / 4-bit weight) + 8-bit embedding21  (`embedding_quantize: "8,0"`; cuts 1143 MB → 741 MB vs the fp32-embedding v1)22- **Export**: executorch 1.4.0 `export_llm`, dynamic shape, max_seq_length 2048, XNNPACK extended_ops23- **Config**: `llm_params/lfm2_5_1_2b_xnnpack_8da4w_e8.yaml`24 25## Verification (2026-08-13)26 27Mac gate (greedy via `native.py`, chat template): correct 2-sentence Rayleigh-scattering28answer, 170.8 tok/s on M-series Mac (reference only). v1 (fp32 embedding) passed 3/329(Paris / Japanese / haiku) with identical quant settings otherwise.30 31iPhone 17 Pro / iOS 27 (ETBench, XNNPACK CPU, default threads), re-measured322026-08-14 on this 8-bit-embedding build:33 34| metric | value |35|--------|-------|36| load | 0.6 s |37| ttft (short prompt) | 0.05-0.06 s |38| decode | **65-86 tok/s** (86 short answer, 65 at 128 tokens) |39 40Outputs correct (Paris; coherent 128-token story). The earlier 1143 MB41fp32-embedding build loaded in 1.6 s and decoded 55-81 tok/s, so quantizing the42embedding table cut both the file and the load time without costing throughput.43 44**Usage note — chat template is required.** This is an instruct model: raw untemplated45text makes it emit `<|im_end|>` immediately (looks like broken generation but is not).46Always wrap prompts as47`<|startoftext|><|im_start|>user\n...<|im_end|>\n<|im_start|>assistant\n`, eos ids [7].48 49## Core ML build (Neural Engine, iOS 18+)50 51`lfm2_5_1_2b_coreml.pte` (2.35 GB, no quantization)52 53The same model on ExecuTorch's Core ML delegate. That path did not run until now: a buffer54written a step after it is read was taken for constant data and handed to the delegate,55Core ML compiled it into a state, and the runtime — told by `take_over_mutable_buffer=False`56that it has none — failed at execute on `layers_N_conv_conv_state`. Fix in57[pytorch/executorch#21979](https://github.com/pytorch/executorch/pull/21979).58 59**The `.pte` needs no patched runtime.** The fix is export-side; this file was verified on60stock executorch 1.4.0 from pip.61 62### Verification (Mac arm64, 2026-08-21)63 64Greedy next-token argmax against `LiquidAI/LFM2.5-1.2B-Instruct` in fp32 eager, five65prompts, every position after the second counted — 27 of 27 agree:66 67| prompt | Core ML | eager |68|---|---|---|69| The capital of France is | ` Paris` | ` Paris` |70| The largest planet in our solar system is | ` Jupiter` | ` Jupiter` |71| Shakespeare wrote a play called Romeo and | ` Juliet` | ` Juliet` |72 73Decode 68 tok/s on the Mac, median of 16 steps after four warm-up steps.74 75**Reset the cache between sequences.** The KV cache persists across `execute` calls, so a76second prompt started at position 0 reads the first one's keys. Load a fresh method per77sequence.78 79### Why no quantization80 81`coreml_quantize: c4w` runs but loses accuracy on this path — measured on the 350M sibling,8221/27 against 27/27 unquantised. The XNNPACK 8da4w file above stays the small build.83