gomyk/jina-v5-h256-distilled-conv

jina-v5-h256-distilled-conv

Lightweight multilingual sentence encoder compressed from `jinaai/jina-embeddings-v5-text-nano` (EuroBERT-210M, 12L/768d) via layer pruning + vocabulary pruning + knowledge distillation with conversation data.

Model Details

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("gomyk/jina-v5-h256-distilled-conv", trust_remote_code=True)
embeddings = model.encode(["Hello world", "How are you?"])
print(embeddings.shape)  # (2, 256)

Training

Stage 1: Architecture Compression

• —Teacher: jinaai/jina-embeddings-v5-text-nano (12 layers, 768d, 128K vocab)
• —Layer pruning: 12 -> 6 layers
• —Dimension reduction: 768d -> 256d
• —Vocabulary pruning: BPE merge-backtracked corpus-based filtering

Stage 2: Knowledge Distillation

• —Method: MSE + Cosine Similarity loss (MSE weight=1.0, Cosine weight=0.5)
• —Data: MTEB task datasets (~1.4M) + conversation data (~19.5M) = ~20.9M texts
• —Optimizer: AdamW (lr=2e-5, weight_decay=0.01)
• —Schedule: Cosine annealing
• —Training: ~2.5 epochs (1.37M global steps), best loss=0.0232
• —Projection: Learnable linear projection 256d -> 768d for distillation

MTEB Evaluation Results

Overall Average: 56.29%

Classification

Clustering

STS (Semantic Textual Similarity)