Modelpublic

LiquidAI/LFM2.5-Encoder-350M

sourceHugging Faceotherupdated 2mo agoView on Hugging Face

145likes24kdownloads

Model Card

<div align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/61b8e2ba285851687028d395/2b08LKpev0DNEk6DlnWkY.png" alt="Liquid AI" style="width: 100%; max-width: 100%; height: auto; display: inline-block; margin-bottom: 0.5em; margin-top: 0.5em;" /> <div style="display: flex; justify-content: center; gap: 0.5em; margin-bottom: 1em;"> <a href="https://playground.liquid.ai/"><strong>Try LFM</strong></a> • <a href="https://docs.liquid.ai/lfm/getting-started/welcome"><strong>Docs</strong></a> • <a href="https://leap.liquid.ai/"><strong>LEAP</strong></a> • <a href="https://discord.com/invite/liquid-ai"><strong>Discord</strong></a> </div> </div>

LFM2.5-Encoder-350M

LFM2.5-Encoder is a family of multilingual bidirectional encoders built on the LFM2 architecture, available in two sizes:

—**LFM2.5-Encoder-230M** — a lightweight encoder for tight latency and memory budgets, punching above its size class.
—LFM2.5-Encoder-350M (this model) — a larger sibling for maximum downstream quality.

Both are masked language models with full bidirectional attention, designed to be fine-tuned into task-specific models (classification, token classification, retrieval, reranking, and semantic similarity) across 15 languages, and to run efficiently on-device.

Find more details about our encoders in our blog post.

Key highlights:

—Top quality for its size. Ahead of every model its size or smaller, and ~5 points above our own retrieval siblings.
—General-purpose. 8k context, strong across NLI, paraphrase, sentiment, and multilingual tasks.
—Fast and on-device. Matches or beats ModernBERT throughput, with a long-context edge on CPU.

[!NOTE] 💻 Demos: We built the demos below from fine-tuned LFM2.5-Encoders. Each one runs in a CPU-only Hugging Face space: - [Zero-shot prompt routing](https://huggingface.co/spaces/LiquidAI/prompt-routing) — define your own routing lanes as free text. The model scores the whole prompt against every lane in one pass. - [Zero-shot policy linting](https://huggingface.co/spaces/LiquidAI/policy-linting) — check text against your company's rules, written as free text. It scores every token against every rule in one pass. - [Spell checking](https://huggingface.co/spaces/LiquidAI/spellchecker) — correct misspellings token by token. - [PII detection](https://huggingface.co/spaces/LiquidAI/pii-detection) — spot and remove 40 kinds of personal information across 16 languages. - [Masked-diffusion text generation](https://huggingface.co/spaces/LiquidAI/masked-diffusion) — bonus: run the encoder as a chatbot that generates text by iteratively unmasking instead of left to right.

📄 Model details

Property	LFM2.5-Encoder-230M	LFM2.5-Encoder-350M
Type	Bidirectional encoder (masked language model)	Bidirectional encoder (masked language model)
Backbone	LFM2	LFM2
Total parameters	~229.7M	~354.5M
Hidden size	1024	1024
Vocabulary size	65,536	65,536
Context length	8,192 tokens	8,192 tokens
License	LFM Open License v1.0	LFM Open License v1.0

Supported languages: English, German, Spanish, French, Italian, Dutch, Polish, Portuguese, Arabic, Hindi, Japanese, Russian, Turkish, Vietnamese, Chinese (15).

Architecture. LFM2.5-Encoder is built on the LFM2 hybrid backbone, which interleaves gated short-convolution blocks with grouped-query attention. For encoder use, the causal mask is replaced with full bidirectional (non-causal) attention and the model is trained with a masked language modeling head. The encoder body is exposed as Lfm2BidirectionalModel; masked-LM loading uses Lfm2BidirectionalForMaskedLM. Both are wired through auto_map and require trust_remote_code=True.

Lfm2BidirectionalForMaskedLM(
  (lfm2): Lfm2BidirectionalModel
  (lm_head): Linear(in_features=1024, out_features=65536, bias=False)
)

Training. LFM2.5-Encoder-350M is adapted from the LFM2 base and trained with a masked language modeling objective on a large multilingual corpus. Pre-training uses a two-stage schedule that extends the context window to up to 8,192 tokens.

We recommend fine-tuning LFM2.5-Encoder-350M for a range of downstream tasks, such as:

—Text classification: sentiment, topic, intent/routing, moderation, and business-text linting.
—Token classification: named-entity recognition, span extraction, and sequence labeling.
—Retrieval and reranking: a backbone for dense embedding or late-interaction (ColBERT-style) retrievers.
—Semantic similarity: STS, paraphrase, and duplicate detection.
—Natural language inference and extractive QA: sentence-pair reasoning and answer-span extraction.

🏃 How to run

Install the latest version of transformers:

bash

pip install -U transformers

Run masked-token prediction:

python

from transformers import AutoModelForMaskedLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("LiquidAI/LFM2.5-Encoder-350M", trust_remote_code=True)
mlm = AutoModelForMaskedLM.from_pretrained("LiquidAI/LFM2.5-Encoder-350M", trust_remote_code=True)

text = f"The capital of France is {tok.mask_token}."
enc = tok(text, return_tensors="pt")
with torch.no_grad():
    logits = mlm(**enc).logits
pos = (enc["input_ids"][0] == tok.mask_token_id).nonzero()[0].item()
print([tok.decode([t]).strip() for t in logits[0, pos].topk(5).indices.tolist()])
# -> ['Paris', 'Strasbourg', 'Paris', 'Lyon', 'Versailles']

For downstream tasks, load the encoder body and attach your own head (classification, token classification, regression, retrieval):

python

from transformers import AutoModel
body = AutoModel.from_pretrained("LiquidAI/LFM2.5-Encoder-350M", trust_remote_code=True)

If your GPU supports it, we recommend using LFM2.5-Encoder-350M with Flash Attention 2 to reach the highest efficiency. To do so, install Flash Attention as follows, then use the model as normal:

bash

pip install flash-attn

📊 Performance

For each benchmark task, we run a full supervised fine-tune and report that fine-tuned model's score. The results below span 14 models across 17 tasks from GLUE, SuperGLUE, and multilingual classification tasks. The full evaluation harness is open-sourced in the `eurobert-repro` repository.

benchmark_ranking

17-task results (avg@5 fresh seeds ± std)

Rank	Model	Params	17-task mean	± std
1	XLM-R XL (3.5B)	3.5B	83.06	±1.16
2	ModernBERT-large (395M)	395M	81.68	±2.49
3	XLM-R large (560M)	560M	81.34	±1.66
4	LFM2.5-Encoder-350M (ours)	350M	81.02	±1.00
5	mDeBERTa-v3 (280M)	280M	80.37	±1.06
6	LFM2.5-Encoder-230M (ours)	230M	79.29	±1.02
7	ModernBERT-base (149M)	149M	78.19	±1.39
8	XLM-R base (280M)	280M	77.46	±1.63
9	EuroBERT-210M	210M	76.87	±2.00
10	mGTE-MLM (305M)	305M	76.53	±1.85
11	LFM2.5-ColBERT-350M	350M	76.18	±1.25
12	EuroBERT-610M	610M	75.87	±2.03
13	LFM2.5-Embedding-350M	350M	75.68	±0.83
14	EuroBERT-2.1B	2.1B	72.19	±5.59

<details> <summary>Click to expand per-task results — all 17 tasks (avg@5 fresh seeds ± std)</summary>

Per-task results — all 17 tasks (avg@5 fresh seeds ± std)

Model	XNLI	PAWS-X	Amazon	MASSIVE	SeaHorse	CoLA*	SST-2*	MRPC*	STS-B*	QQP*	MNLI*	QNLI*	RTE*	BoolQ*	CB*	WiC*	WSC*	ALL
XLM-R XL (3.5B)	87.12±0.45	93.30±0.41	62.29±0.05	88.21±0.32	59.51±3.52	84.58±1.18	95.69±0.30	87.65±1.69	90.20±0.93	91.72±0.07	90.09±0.11	94.47±0.28	82.38±3.51	83.70±0.24	89.88±3.72	66.55±1.37	64.62±1.58	83.06
ModernBERT-large (395M)	81.76±0.38	92.46±0.18	60.42±0.15	85.65±0.94	40.20±17.18	83.37±0.20	96.10±0.53	88.14±1.79	92.16±0.24	91.81±0.13	90.65±0.18	94.36±0.10	81.59±4.92	81.68±2.34	88.21±3.24	70.16±2.57	69.81±7.18	81.68
XLM-R large (560M)	84.69±0.59	93.23±0.78	61.58±0.13	88.50±0.17	56.12±2.31	83.34±1.75	93.83±1.04	88.77±2.15	91.35±0.23	90.48±0.23	88.29±0.07	93.08±0.23	80.79±3.14	80.54±0.79	78.21±8.69	66.24±5.41	63.65±0.43	81.34
LFM2.5-Encoder-350M (ours)	79.82±0.29	91.53±0.73	60.57±0.09	85.70±0.13	54.96±0.41	84.43±0.81	95.11±0.26	87.21±1.86	91.59±0.05	92.08±0.10	89.03±0.17	93.97±0.23	75.23±3.84	81.52±0.69	83.21±2.04	69.66±2.23	61.73±3.15	81.02
mDeBERTa-v3 (280M)	83.01±0.47	92.59±0.39	60.62±0.31	87.64±0.51	54.97±2.04	83.91±1.03	92.41±0.96	85.39±2.59	89.87±0.22	90.23±0.15	86.30±0.18	91.99±0.35	69.75±2.03	78.29±1.39	88.21±2.40	67.71±3.05	63.46±0.00	80.37
LFM2.5-Encoder-230M (ours)	77.63±0.31	90.86±0.24	59.97±0.21	85.52±0.69	54.61±0.62	81.42±1.56	94.08±0.37	80.20±2.66	90.99±0.12	91.71±0.07	87.98±0.22	92.96±0.37	67.29±1.97	76.54±1.01	83.21±4.48	70.31±1.34	62.69±1.05	79.29
ModernBERT-base (149M)	76.64±0.31	92.17±0.15	58.98±0.11	85.32±0.21	45.19±1.72	83.07±1.93	94.79±0.52	84.46±2.70	90.75±0.14	91.23±0.11	88.68±0.17	93.04±0.36	58.70±1.94	74.78±4.10	81.07±6.75	66.90±2.39	63.46±0.00	78.19
XLM-R base (280M)	78.20±0.80	91.36±0.41	60.01±0.12	87.47±0.49	51.08±3.75	81.17±1.07	91.97±0.18	86.47±0.76	88.27±0.36	89.21±0.04	83.07±0.21	90.17±0.32	62.60±7.03	71.43±1.64	79.64±7.53	61.25±3.00	63.46±0.00	77.46
EuroBERT-210M	80.83±0.35	91.94±0.31	59.94±0.16	86.36±0.67	45.16±16.60	72.75±1.30	90.64±0.92	80.74±2.99	89.29±0.23	90.75±0.09	85.63±0.28	91.49±0.27	54.95±2.56	71.68±2.35	86.79±2.40	64.64±2.01	63.27±0.43	76.87
mGTE-MLM (305M)	80.32±0.20	91.73±0.26	60.26±0.10	87.79±0.20	51.58±1.31	75.44±4.66	91.19±1.00	86.32±1.48	87.77±0.64	89.82±0.09	84.14±0.15	90.94±0.40	58.34±3.09	69.32±3.72	73.21±6.80	59.34±7.41	63.46±0.00	76.53
LFM2.5-ColBERT-350M	78.77±0.47	89.74±0.44	59.92±0.13	86.65±0.17	47.95±1.13	71.06±1.06	90.94±0.78	73.43±7.22	89.38±0.28	91.11±0.14	84.70±0.23	90.66±0.18	59.13±2.30	74.25±1.61	81.79±2.93	62.04±2.12	63.46±0.00	76.18
EuroBERT-610M	84.61±0.34	91.84±0.94	60.64±0.08	86.03±0.99	12.91±8.05	70.60±2.12	92.52±0.66	85.20±1.34	89.82±0.22	91.13±0.09	87.95±0.19	92.57±0.36	59.28±7.93	76.86±1.55	85.71±4.37	58.71±5.30	63.46±0.00	75.87
LFM2.5-Embedding-350M	78.59±0.11	89.13±0.63	60.47±0.13	87.03±0.21	50.19±0.90	72.54±0.68	91.70±0.69	77.45±1.31	89.38±0.09	91.14±0.13	84.70±0.12	90.62±0.50	55.38±1.74	70.17±1.99	71.43±3.57	63.10±1.28	63.46±0.00	75.68
EuroBERT-2.1B	70.52±14.22	92.34±0.19	60.45±0.70	85.40±1.36	6.84±6.81	68.99±0.67	92.50±1.03	82.94±3.10	66.44±32.36	91.03±0.29	81.56±16.62	93.56±0.25	53.29±0.79	77.23±6.77	82.86±5.14	57.90±4.75	63.46±0.00	72.19

* = dev split (GLUE/SuperGLUE test labels hidden). The 5 multilingual columns are labeled test. SeaHorse & STS-B are Spearman×100. All other tasks are accuracy. </details>

Inference speed

The LFM2 backbone was built for fast inference, and the encoders inherit it. While ModernBERT-base is faster at short sequences in Apple GPU inputs, LFM2.5-Encoders overtake it as inputs grow. At long input sequences of 8k on CPU, the encoders run 3.3× faster than ModernBERT-base.

inference_perf_cpu

inference_perf_gpu

🔧 Fine-tuning

LFM2.5-Encoder-350M follows standard BERT-style fine-tuning. Attach a task head to the encoder body and train end-to-end. Suggested starting points (tune per task):

Hyperparameter	Suggested range
Learning rate	1e-5 – 5e-5
Warmup ratio	0.1
Weight decay	0.1
Epochs	3 – 20 (early stopping, patience 3)
Precision	bf16 autocast (fp32 master weights)

📬 Contact

—Got questions or want to connect? Join our Discord community
—If you are interested in custom solutions with edge deployment, please contact our sales team.

Citation

bibtex

@article{liquidAI2026Encoders,
  author = {Liquid AI},
  title = {LFM2.5-Encoders: Fast at Long Context, Even on CPU},
  journal = {Liquid AI Blog},
  year = {2026},
  note = {www.liquid.ai/blog/lfm2-5-encoders},
}