oxfrug/bge-m3-swedish-superlim

bge-m3-swedish-superlim

Swedish contrastive fine-tune of `BAAI/bge-m3`. Beats `KBLab/sentence-bert-swedish-cased` on both SuperLim-2 tasks that card published: SweParaphrase (STS) and SweFAQ (in-category retrieval).

Result: vs vanilla BGE-M3, this checkpoint improves SuperLim SweParaphrase Spearman by +2.3% (0.8165 → 0.8354) and SweFAQ in-category accuracy by +7.6% (0.6055 → 0.6514). Clean Swedish retrieval (SwednRetrieval nDCG@10, no train overlap) is +6.5% (0.7335 → 0.7809). Nordic MTEB mean is only +1.5%; Swedish classification dropped vs the base (SweRec −0.047, SwedishSentiment −0.028). Not a “multilingual hold.”

Related: `oxfrug/qwen3-embedding-0.6b-swedish-superlim` — Qwen-family 0.6B on the same protocol.

Model details

The 768-d cut keeps the STS win and lifts SweFAQ to the Qwen3-Embedding-8B zero-shot number.

Quickstart

pip install -U sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("oxfrug/bge-m3-swedish-superlim")
emb = model.encode(["Ett plan lyfter.", "Ett flygplan lyfter."], normalize_embeddings=True)
print(float(emb[0] @ emb[1]))

Retrieval at the recommended 768-d cut (query prefix + Matryoshka truncate):

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("oxfrug/bge-m3-swedish-superlim")
QUERY = "Represent this sentence for searching relevant passages: "

def embed(texts, query=False, dim=768):
    texts = [QUERY + t if query else t for t in texts]
    vec = model.encode(texts, normalize_embeddings=True)
    if dim and vec.shape[1] > dim:
        vec = vec[:, :dim]
        vec /= np.clip(np.linalg.norm(vec, axis=1, keepdims=True), 1e-12, None)
    return vec

q = embed(["När får jag föräldrapenning?"], query=True)
d = embed(["Du kan ansöka på Mina sidor."], query=False)
print(float(q[0] @ d[0]))

STS / clustering: no prefix. Retrieval: prefix the query only.

Benchmarks

All “this” numbers below are @768 (first 768 dims, re-L2). Base is vanilla BAAI/bge-m3 on the same protocol. Δ is this − base.

SuperLim-2 (held-out test)

Spearman of cosine vs human 0–5 on SweParaphrase (n=1378). SweFAQ is in-category accuracy (n=109). Same frozen sbx/superlim-2 splits as the KBLab re-run.

SweParaphrase test was never in train. SweFAQ used the official train split (plus Swedish healthcare Q–A). Do not call SweFAQ zero-shot. SweFAQ n=109 is +8 correct vs the KBLab re-run; treat it as a small test.

Swedish tasks — MTEB (Scandinavian, v1)

mteb 2.19.3, max_seq_length=512 (KBLab 256). Official SweFAQ score is nDCG@10, not SuperLim in-category accuracy.

† Supervised for this model. SwednRetrieval is the number to quote for Swedish retrieval generalization.

KBLab on the same nine tasks is in the dump; SV mean 0.6132. This mix beats KBLab on the Swedish mean, but that is not the before/after vs BGE.

Nordic category means (27/28 tasks)

DKHateClassification gated, skipped.

The mix is a retrieval specialist. Classification and a bit of bitext were spent to buy SweFAQ / Swedn. Not a general “best Nordic embedder.”

Raw dumps: `superlim/test.json`, `mteb/MTEB_Scandinavian_v1/`. These files document the run; they do not put the model on the official MTEB leaderboard.

Why it moved

Training is MNRL on high-similarity Swedish pairs (SweParaphrase ≥ 4) plus FAQ / healthcare query–passage pairs with the BGE retrieval prefix. That is the same geometry SuperLim SweParaphrase and SweFAQ / SwednRetrieval reward, so those go up. Label-driven classification (SweRec, SwedishSentiment) was not in the loss; the contrastive mix pulled those embeddings around and they dropped vs the already-strong multilingual BGE baseline. SwednClusteringS2S rose; P2P did not — short-text clustering is not the same as FAQ retrieval.

How it was made

Dataset composition and licensing

Weights are MIT (same as BGE-M3). Training text is not MIT. SuperLim pieces are CC BY 4.0 (Språkbanken). Healthcare Q–A is an extra in-house mix; source file / license was not checked into this repo — treat as unknown, do not redistribute that split from here.

SuperLim dev/test strings were blocked. SweFAQ test is therefore not zero-shot. Swedn / SweRec / Dalaj / SwedishSentiment were not in train.

Hub dataset: `sbx/superlim-2` (sweparaphrase, swenli, swefaq).

Training configuration (what is known)

Not in this repo (original llm-lab training tree is not published here): learning rate, batch size, warmup, max sequence length during train, seed, exact pair counts after filtering, wall-clock, GPU SKU. Do not invent those. 4-bit NF4 after the fact gives the STS win back (0.820); ship bf16.

Reproducing evaluation

MTEB Scandinavian as on this card:

pip install "mteb==2.19.3" sentence-transformers
python - <<'PY'
import mteb
from sentence_transformers import SentenceTransformer

class BgeSv768(SentenceTransformer):
    """First-768 Matryoshka cut + re-L2, BGE query prefix on retrieval."""
    pass

# Card numbers used max_seq_length=512, encode batch_size=32,
# normalize_embeddings=True, skipped DKHateClassification.
# Raw payloads: mteb/MTEB_Scandinavian_v1/ on this repo.
print("Use the JSON dump to match the card; a fresh mteb run needs the same wrapper.")
PY

The JSON under `mteb/MTEB_Scandinavian_v1/` is the actual mteb.evaluate output (mteb 2.19.3, torch 2.13, sentence-transformers 5.7, transformers 5.14.1). SuperLim protocol: Spearman of cosine vs 0–5 on SweParaphrase-test; SweFAQ in-category accuracy on 109 items, query prefix on, dim 768. Dump: `superlim/test.json`.

A bit-identical training script is not in this repository.

Intended use

Swedish STS, clustering, and retrieval. Recommended embedding size is 768. Prefer this over vanilla BGE-M3 when SweFAQ / Swedn-style retrieval matters. Prefer vanilla BGE-M3 when SweRec / SwedishSentiment-style classification is the job.

Limits

• —568M / ~1.1 GB bf16. Slower to encode than 110M KBLab.
• —SweFAQ test is 109 items. @768 is +8 correct vs the KBLab re-run — noisy.
• —SuperLim claim is SweParaphrase + SweFAQ only.
• —Known failure vs base BGE-M3: SweRecClassification (−0.047), SwedishSentimentClassification (−0.028), Nordic classification mean (−0.016), SwednClusteringP2P (−0.010), MassiveIntent sv (−0.009).
• —SweFAQ nDCG@10 on MTEB is supervised. Quote SwednRetrieval for generalization.
• —4-bit NF4 gives the STS win back (0.820). Ship bf16.
• —Healthcare Q–A license unknown; SuperLim train licenses are CC BY 4.0 — the weights stay MIT, the data does not.

License

MIT, same as BGE-M3. Fine-tune by oxfrug. SuperLim training data is CC BY 4.0 (Språkbanken); cite SuperLim if you redistribute derivatives of that split.

Citation

@misc{bge-m3,
  title={BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation},
  author={Chen, Jianlv and Xiao, Shitao and Zhang, Peitian and Luo, Kun and Lian, Defu and Liu, Zheng},
  year={2024},
  eprint={2402.03216},
  archivePrefix={arXiv}
}

@inproceedings{berdicevskis-etal-2023-superlim,
  title = {Superlim: A {S}wedish Language Understanding Evaluation Benchmark},
  author = {Berdicevskis, Aleksandrs and Bouma, Gerlof and Kurtz, Robin and Morger, Felix and {\"O}hman, Joey and Adesam, Yvonne and Borin, Lars and Dann{\'e}lls, Dana and Forsberg, Markus and Isbister, Tim and Lindahl, Anna and Malmsten, Martin and Rekathati, Faton and Sahlgren, Magnus and Volodina, Elena and B{\"o}rjeson, Love and Hengchen, Simon and Tahmasebi, Nina},
  booktitle = {Proceedings of EMNLP 2023},
  year = {2023}
}