LingoIITGN/FlowVec-v1

FlowVec-v1

A multilingual text embedding model for the scheduled Indic languages, based on Qwen/Qwen3-Embedding-8B via a 3-stage LoRA recipe with frozen-teacher preservation.

The training objective combines supervised-contrastive learning with a relational distillation loss against a frozen copy of the base model, so task-specific geometry can be reshaped in one domain while the base model's general-purpose behaviour is held in place elsewhere.

• —Base model: Qwen/Qwen3-Embedding-8B
• —Training: LoRA (r=64, alpha=128), merged into base for release
• —Pooling: last-token, L2-normalized
• —Embedding dimension: 4096
• —Max sequence length: 512
• —License: Apache 2.0 (inherited from base model)

Evaluation

Evaluated on MTEB(Indic, v1) using the official mteb package.

Mean (TaskType): 80.61 · Mean (Task, 20 tasks): 78.05

Per-category results

Excluding SIB200ClusteringS2S — the one task with training-data overlap — the mean over the remaining six task types is 79.45.

Per-task results

Full per-task JSON results are in the MTEB results repository.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("<HF_REPO_ID>")

# Retrieval -- asymmetric: instruction on the query side only
queries = [
    "भारत की राजधानी क्या है?",
    "চা কীভাবে বানানো হয়?",
]
documents = [
    "नई दिल्ली भारत की राजधानी है।",
    "গরম জলে চা পাতা ভিজিয়ে চা তৈরি করা হয়।",
]

query_embs = model.encode(queries, prompt_name="Retrieval-query", normalize_embeddings=True)
doc_embs   = model.encode(documents, prompt_name="Retrieval-document", normalize_embeddings=True)
print(model.similarity(query_embs, doc_embs))

# Symmetric tasks -- same instruction on both sides
emb = model.encode(
    ["भारतीय अंतरिक्ष अनुसंधान संगठन ने नया उपग्रह प्रक्षेपित किया।"],
    prompt_name="Clustering",
    normalize_embeddings=True,
)

Prompts matter substantially: on the clustering task the gap between the shipped prompt and no prompt is roughly 42 points. Pass a prompt_name whenever one applies.

Available names include the task types (Clustering, Classification, PairClassification, BitextMining, STS, Retrieval-query, Retrieval-document) and per-task overrides that select a more specific instruction (SIB200ClusteringS2S, XNLI, BengaliSentimentAnalysis, MTOPIntentClassification, …). Classification prompts are routed by task kind — sentiment, topic, intent, or none. See config_sentence_transformers.json for the full table.

Training procedure

Three LoRA stages, in every stage a frozen full-precision copy of the base model acts as a preservation teacher through two terms: a relational loss (MSE between the student and teacher pairwise-cosine Gram matrices) and a pointwise cosine loss. The contrastive weight stays modest while the teacher weight rises across stages, which is what prevents catastrophic forgetting of the base model's general capabilities.

Precision: bf16 mixed precision throughout, on 2×H200.

Training data

All sources use training splits only.

• —Retrieval / bitext (Stage 1): Samanantar (ai4bharat), plus curated retrieval pairs with mined hard negatives.
• —Classification / NLI (Stage 2): IndicXNLI (entailment as positive, contradiction as hard negative), MASSIVE intent classification, sentiment data, and language-identification triplets constructed from parallel bitext with same-script hard negatives.
• —Clustering (Stage 3): SIB-200 topic pairs across 18 Indic languages, each with mined same-language, different-category hard negatives, mixed with translation and NLI replay batches. Languages absent from MTEB(Indic, v1) were dropped and Kashmiri-Devanagari added, so the Stage-3 language set matches the evaluation set.

Data was sanitized to remove control characters and validated for JSONL round-tripping before training.

Evaluation caveats

Two things affect how the scores above should be read.

Training-data overlap. MTEB treats a model as zero-shot only if it was not trained on any split of the dataset a task derives from. By that definition this model is not zero-shot on two of twenty tasks:

That is a ~90% zero-shot score. The clustering figure is an in-domain result and should not be read as general clustering ability. The other eighteen tasks — all of Retrieval, Reranking, BitextMining and STS — have no overlap.

Prompt selection. Prompts were chosen by measuring candidates on the benchmark itself rather than written blind. Shipping task-specific instructions is standard for instruction-tuned embedding models, but this procedure makes the reported scores optimistic relative to prompts chosen a priori.

Model architecture

• —Base: Qwen3-Embedding-8B (decoder-only transformer, 4096 hidden dim)
• —Pooling: last-token (matches the Qwen3 base convention)
• —Normalization: L2, applied on the pooled vector
• —LoRA targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• —LoRA rank: 64, alpha: 128, dropout: 0.05
• —Merged: yes — the adapter is merged into the base weights for release, so no peft dependency is required at inference

Instruction format

Instruct: {task_description}\nQuery: {query_text}

Retrieval and reranking are asymmetric: documents are encoded without an instruction prefix. Clustering, classification, pair-classification, bitext and STS apply the same instruction to both sides. Prompt strings live in config_sentence_transformers.json and are applied by sentence-transformers when prompt_name is passed to encode().