gomyk/me5s-student-me5s_compressed_distilled_v2

mE5-small Compressed & Distilled (v2)

A 4-layer, 384d, 15K vocab compressed student of intfloat/multilingual-e5-small, distilled on ~1.4M MTEB sentences for 20 epochs.

Model Summary

MTEB Benchmark Results

STS (Semantic Textual Similarity)

Note: STS17 is a cross-lingual task heavily impacted by vocab pruning (250K->15K tokens). Excluding STS17: Teacher 0.7893 vs Student 0.7496 (delta -0.040, 95.0% retention).

Classification

Clustering

Overall Summary

9x smaller model size (450MB -> 51MB), 89.5% overall performance retention. Classification exceeds teacher performance. STS retains 95% (excluding cross-lingual STS17).

Compression Pipeline

1. 1.Layer pruning: 12 -> 4 layers (uniform spacing: layers 0, 4, 8, 11)
2. 2.Vocabulary pruning: 250,037 -> 15,168 tokens (corpus-seen tokens only)
3. 3.Distillation: MSE + Cosine loss on ~1.4M MTEB corpus sentences, 20 epochs
4. 4.Best distillation loss: 0.0096

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("gomyk/me5s-student-me5s_compressed_distilled_v2")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)  # [3, 384]

similarities = model.similarity(embeddings, embeddings)
print(similarities)

Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertModel'})
       - 4 layers, 384d hidden, 12 attention heads
       - 15,168 vocab (XLMRobertaTokenizer, pruned)
  (1): Pooling(mean_tokens)
)

Training Details

• —Teacher: intfloat/multilingual-e5-small (12L, 384d, 250K vocab)
• —Distillation loss: MSE + 0.5 * Cosine similarity loss
• —Corpus: ~1.4M sentences from MTEB Classification/Clustering/STS/NLI datasets
• —Epochs: 20 (best loss: 0.0096)
• —Batch size: 32, LR: 2e-5, Cosine annealing scheduler
• —Framework: PyTorch 2.10, Transformers 4.56, Sentence-Transformers 5.3

Limitations

• —Cross-lingual STS (STS17) significantly degraded due to aggressive vocab pruning
• —Clustering performance drops more than other categories
• —Optimized for English; multilingual coverage is reduced by vocab pruning