andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir-langmoe

apertus-v1.5-swiss-embed-4.9b-bidir-langmoe

A Sentence Transformers retrieval model adapted from `andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir` by replacing the parent's final dense embedding projection with a learned sentence-level SparseMixtureOfExperts projection.

Architecture

existing Sentence Transformers embedding parent
    ↓
all modules before the parent's final Dense
    ↓
SparseMixtureOfExperts
    ├── learned linear router
    ├── softmax
    ├── Top-2
    └── weighted selected expert projections
    ↓
parent Normalize

This Swiss experiment uses 7 experts labelled ['de', 'en', 'fr', 'it', 'rm', 'gsw', 'shared']. The labels are experimental metadata; the MoE module itself contains no language-specific logic.

No router temperature is used.

Sentence Transformers usage

SparseMixtureOfExperts is currently exported as a repository-local candidate module. Until an equivalent module is registered upstream in Sentence Transformers, loading this experimental artifact requires trust_remote_code=True:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "andreasmartin/apertus-v1.5-swiss-embed-4.9b-bidir-langmoe",
    trust_remote_code=True,
)

queries = model.encode_query([
    "Welche Aufgaben hat der Bundesrat?"
])

documents = model.encode_document([
    "Le Conseil fédéral est l'autorité exécutive suprême de la Confédération suisse.",
    "Der Nationalrat ist die grosse Kammer der Bundesversammlung.",
])

scores = model.similarity(queries, documents)
print(scores)

If SparseMixtureOfExperts becomes a registered Sentence Transformers module, the same conceptual architecture can be serialized without repository-local Python code and loaded without trust_remote_code=True.

Parent adaptation

The parent was already an embedding model. This notebook does not create or modify a native backbone-level MoE LLM.

Training

The parent modules before the final Dense projection were frozen and their sentence representations cached. The original Dense projection initialized every expert, so the model starts from the parent's embedding transformation before experts specialize.

The retrieval objective is combined with:

• —optional weak router supervision from the declared language labels;
• —a load-balancing regularizer.

The declared language is not required at inference. Routing is computed from the sentence representation itself.

Training data:

• —`wikimedia/wikipedia`
• —`eljuanina/VotingBooklets-v1`
• —`ZurichNLP/SwissGov-RSD`

eljuanina/VotingBooklets-Diamond-v1 is held-out evaluation-only.

Actual training set: 31,483 triplets.

See training_metadata.json for the exact training configuration, router_diagnostics.json for held-out routing counts, and data_manifest.json for immutable dataset revisions and processed-file hashes.

Internal retrieval diagnostics

These metrics are internal development diagnostics, not MTEB/MMTEB benchmark claims.

Monolingual evaluation queries: 480 Cross-lingual evaluation queries: 480

Router diagnostics

Recorded shared-expert Top-k selections in the held-out diagnostic: 15.

Limitations

• —The adapter currently targets existing Sentence Transformers parents ending in Dense -> Normalize.
• —Sparse routing occurs only at sentence-embedding projection level.
• —Language labels are weak supervision for this experiment, not an architectural requirement.
• —External MTEB/MMTEB/MIRACL and Swiss/domain-specific evaluation is required for comparative claims.
• —The MoE head was trained on cached representations up to 512 tokens. The packaged parent retains 1024-token inference capability, but quality beyond the MoE training context length has not yet been separately validated.