Modelpublic

Lzvick/bge-m3-ir-research-lora-v1

sourceHugging Facemitupdated 3mo agoView on Hugging Face

0likes13downloads

Model Card

bge-m3-ir-research-lora-v1

LoRA fine-tune of `BAAI/bge-m3` for dense retrieval over information-retrieval research literature (BM25, DPR, ColBERT, RAG, MS MARCO, BEIR). Weights below are the merged model (base + adapter) — no peft needed to load it.

Usage

python

from FlagEmbedding import BGEM3FlagModel

model = BGEM3FlagModel("Lzvick/bge-m3-ir-research-lora-v1", use_fp16=True)
output = model.encode(
    ["What is dense passage retrieval?"],
    return_dense=True,
    return_sparse=True,
    return_colbert_vecs=False,
)

Training

Setting	Value
Base model	`BAAI/bge-m3`
Hardware	Single consumer GPU, 12GB VRAM
Method	LoRA (`r=16`, `alpha=32`, `dropout=0.1`, targets: `query,key,value`)
Trainable params	2,359,296 / 570,114,048 (0.41%)
Epochs	3
Effective batch size	128 (4 per-device × 32 grad-accum)
Learning rate	1e-5
Query / passage max len	256 / 256
Train group size	8 (1 positive + 7 negatives per query)
Precision	bf16
Train set	2,072 examples

Evaluation

Sanity check on 109 held-out validation examples: mean cosine-similarity margin between query→positive and query→negative pairs.

Model	Mean pos-neg margin
Baseline (`BAAI/bge-m3`)	0.0849
Fine-tuned (this model)	0.0861

This is a coarse sanity check, not a retrieval benchmark (no nDCG/Recall/MRR test-set evaluation has been run yet). Treat this as an early checkpoint from a short run (~51 optimizer steps), not a fully validated release.