Yigit-Karaman/turkish_focused-multilingual-e5-small

turkish_focused-multilingual-e5-small (V4 LoRA)

This is a domain adapted embedding model based on intfloat/multilingual-e5-small. It has been fine-tuned using Low-Rank Adaptation (LoRA) to significantly boost Turkish-language retrieval and cross-lingual alignment while strictly preserving the base model's English scientific and factoid retrieval capabilities.

🚀 Architecture & Training

To avoid catastrophic forgetting of pre-trained weights, this model utilizes Low-Rank Adaptation (LoRA) via PEFT. Only the attention matrices (query, key, value, dense) were adapted, keeping base representations stable.

• —Base Model: intfloat/multilingual-e5-small
• —LoRA Config: $r=16$, $\alpha=32$, dropout $= 0.05$
• —Epochs: 1.5
• —Batch Size: 160
• —Learning Rate: 3e-5 (warmup: 281 steps)
• —Hardware: Local NVIDIA GeForce RTX 4060 (8GB VRAM)

📚 Training Dataset Composition (300,000 Pairs)

The model was trained on a balanced ~300k-pair corpus designed to blend general QA, instruction-following, and specialized Turkish retrieval:

• —English Retention Corpus (100,000 pairs):
• —rajpurkar/squad
• —yahma/alpaca-cleaned
• —databricks/databricks-dolly-15k
• —Turkish General Corpus (150,000 pairs):
• —boun-tabi/squad_tr
• —merve/turkish_instructions
• —atasoglu/databricks-dolly-15k-tr
• —Targeted Turkish Corpus (50,000 pairs):
• —30,000 pairs: trmteb/turkish_embedding_model_training_data
• —10,000 pairs: AhiskaAI/Ahiska-Turkish-Language-Dataset (passages paired with synthetic queries generated via gpt-5.6-luna and local LLM inference)
• —10,000 pairs: NumanKaanKaratas/turkish-sentences (passages paired with synthetic queries generated via gpt-5.6-luna and local LLM inference)

📊 MTEB Benchmark Evaluation

Evaluated across 11 tasks comparing the merged LoRA model against the original intfloat/multilingual-e5-small base:

💻 Usage

The PEFT adapter matrices have been fused directly into the base weights (merge_and_unload()), allowing direct usage via sentence-transformers without needing peft:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Yigit-Karaman/turkish_focused-multilingual-e5-small")

# E5 models require query/passage prefixes
query = "query: Bilgi erişimi modellerinde LoRA nasıl çalışır?"
doc = "passage: Düşük dereceli uyarlama (LoRA), temel ağırlıkları dondurup dikkat katmanlarına eğitilebilir matrisler ekleyerek çalışır."

embeddings = model.encode([query, doc], normalize_embeddings=True)
similarity = embeddings[0] @ embeddings[1]

print(f"Cosine Similarity: {similarity:.4f}")