Vladimirlv/ru-promptriever-qwen3-1.7b

ru-Promptriever-Qwen3-1.7B

   

Overview

Standard dense retrieval models score query–passage pairs using a single semantic similarity signal, giving users no control over what "relevant" means beyond keyword choice. Promptriever (Weller et al., 2024) introduced per-instance natural language instructions that dynamically redefine relevance — a capability previously limited to generative LLMs.

ru-Promptriever extends this paradigm to Russian:

• —Architecture: Qwen3-based causal LM fine-tuned as a bi-encoder with LoRA + GradCache
• —Pooling: last-token (EOS) pooling, same as the original Promptriever
• —Key training signal: instruction negatives — passages that are topically relevant to the query but violate the instruction constraint

Model Family

This Model

This is a scaling experiment to test instruction-aware retrieval at the 1.7B parameter scale. The model was trained from Qwen3-1.7B on the same dataset as the 0.6B variant for a fair comparison.

Under the current harmonized evaluation protocol, the 1.7B model obtains 12.97 p-MRR and 0.502 nDCG@20 on mFollowIR-RU. This shows that instruction-following retrieval is learnable at this scale, while the 4B model achieves stronger overall results.

Note: For best performance, use the 4B model instead.

Evaluation Results

mFollowIR-RU

Russian split of mFollowIR — multilingual instruction-following retrieval using TREC NeuCLIR narratives as instructions.

p-MRR (Pairwise Mean Reciprocal Rank, ×100) is the primary instruction-following metric — higher means the model correctly adjusts rankings when instructions change. nDCG@20 measures standard retrieval quality.

All rows use the official MTEB 2.10.5 candidate pools and each model's native preprocessing. The table reports point estimates; no pairwise significance claim is made for the 1.7B model.

Additional Benchmarks

These values use the repository's selected step-320 adapter and the corrected MTEB 2.10.5 protocol. FollowIR retrieval columns use the original-instruction qrels; legacy published retrieval values from older FollowIR implementations are not directly comparable.

Usage

Basic Retrieval (no instruction)

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch.nn.functional as F

model_name = "Vladimirlv/ru-promptriever-qwen3-1.7b"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

def encode(texts: list[str], max_length: int = 512) -> torch.Tensor:
    """Encode texts using last-token (EOS) pooling."""
    inputs = tokenizer(
        texts,
        padding=True,
        truncation=True,
        max_length=max_length,
        return_tensors="pt",
    ).to(model.device)

    with torch.no_grad():
        # Bypass lm_head to get post-norm hidden states
        original_lm_head = model.lm_head
        model.lm_head = torch.nn.Identity()
        outputs = model(**inputs, use_cache=False, return_dict=True)
        model.lm_head = original_lm_head

    # EOS pooling: take embedding at last non-padding token
    seq_len = inputs["attention_mask"].sum(dim=1) - 1
    embeddings = outputs.logits[torch.arange(len(texts)), seq_len]
    return F.normalize(embeddings, p=2, dim=1)


query = "Когда была основана Москва?"
passages = [
    "Москва была основана в 1147 году князем Юрием Долгоруким.",
    "Санкт-Петербург был основан Петром I в 1703 году.",
]

q_emb = encode([query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()
print(scores)

Instruction-Following Retrieval

# Append the instruction directly to the query (same format as training)
instruction = "Найди документ, в котором упоминается конкретная дата основания города."
instructed_query = f"{query} {instruction}"

q_emb = encode([instructed_query])
p_emb = encode(passages)
scores = (q_emb @ p_emb.T).squeeze()

Using with sentence-transformers

This model is not compatible with sentence-transformers out of the box due to the custom EOS pooling. Use the snippet above directly with transformers.

Model Details

Training Data

The model was trained on the same dataset as the 0.6B variant for a fair comparison:

1. 1.Russian real data (~11k) — from MIRACL and MrTyDi retrieval datasets
2. 2.Russian synthetic data (~30k) — instruction-augmented and standard pairs from ru-promptriever-dataset

Intended Use

• —Research on model scaling for instruction-following retrieval
• —Benchmarking mid-size models for instruction-aware retrieval tasks

Out-of-Scope

• —Production retrieval systems (use the 4B model instead)
• —Commercial applications (see License below)

Limitations

• —Reduced capacity: The 1.7B model learns positive instruction-following but has lower retrieval and instruction-following point estimates than the 4B model.
• —MS MARCO origin: Training corpus derives from machine-translated English web passages.
• —Synthetic-data quality: A blinded two-author audit of 64 final instances found 95.2% positive-document compliance and 85.2% negative non-compliance with the complete query-instruction condition (79.3% agreement; Cohen's κ = 0.57). The audit is encouraging but small, and imperfect synthetic examples may remain.

License

This model is released under CC BY-NC 4.0 (Creative Commons Attribution–NonCommercial 4.0 International).

The non-commercial restriction is inherited from the upstream MS MARCO license (Microsoft Research License — non-commercial use only), which governs the training corpus.

Citation

If you use this model, please cite the original Promptriever paper:

@article{weller2024promptriever,
  title   = {Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models},
  author  = {Weller, Orion and Van Durme, Benjamin and Lawrie, Dawn and
             Paranjape, Ashwin and Zhang, Yuhao and Hessel, Jack},
  journal = {arXiv preprint arXiv:2409.11136},
  year    = {2024}
}