BCCard/MoAI-Embedding-4B

1---2language:3- ko4license: apache-2.05library_name: sentence-transformers6pipeline_tag: sentence-similarity7base_model: Qwen/Qwen3-Embedding-4B8tags:9- sentence-transformers10- feature-extraction11- sentence-similarity12- text-embedding13- information-retrieval14- korean15- finance16- lora17- peft18datasets:19- BCCard/BCAI-Finance-Kor-Embedding-Triplet20- BCCard/BCAI-Finance-Kor-Embedding-Pair21metrics:22- ndcg23- mrr24- recall25---26 27# 1. Overview28A Korean text-embedding model for the **BC Card domain**, built by LoRA fine-tuning29[`Qwen/Qwen3-Embedding-4B`](https://huggingface.co/Qwen/Qwen3-Embedding-4B) on BC Card in-domain data (personal / merchant / corporate / VIP). It is intended as the **retriever (bi-encoder)** stage of a BC Card RAG pipeline.30 31This is the **4B-scale** sibling of [`BCCard/MoAI-Embedding-0.6B`](https://huggingface.co/BCCard/MoAI-Embedding-0.6B) — a larger-capacity variant for higher retrieval quality at the cost of compute/latency.32 33On a held-out in-domain test set it improves **NDCG@10 by +6.1%** and **Accuracy@1 by +8.9%** over the base `Qwen3-Embedding-4B` (full metrics in §2.3).34 35## 1.1. TL;DR36* **Base model**: [`Qwen/Qwen3-Embedding-4B`](https://huggingface.co/Qwen/Qwen3-Embedding-4B) — 36 layers, hidden 2560, last-token pooling, instruction-aware37* **Domain / Language**: Finance (BC Card — personal / merchant / corporate / VIP) / Korean38* **Task**: Query-document retrieval (QA search, document similarity), RAG retriever39* **Method**: PEFT (LoRA) + Multiple Negatives Ranking (contrastive)40* **Format**: merged standalone (LoRA fused into base; loads with `sentence-transformers`, no `peft`)41* **Embedding dimension**: 2560 · **Max sequence length**: 1024 · **Similarity**: cosine (outputs are L2-normalized)42* **Intended use**43  - In-house **BC Card-domain RAG retriever** (Top-K candidate retrieval)44  - QA search, document-similarity scoring45 46## 1.2. Usage47 48The model was trained with an **instruction prefix on the query side only** (documents get no49instruction). Inject the same instruction at inference so query/document encoding matches training.50 51```python52from sentence_transformers import SentenceTransformer53 54model = SentenceTransformer("BCCard/MoAI-Embedding-4B")55 56# Query-side instruction (identical to training) - prepend to every query at inference time57QUERY_INSTRUCTION = "Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: "58 59queries = ["BC카드 연회비는 어떻게 되나요?"]60documents = [61    "BC카드 연회비는 카드 종류와 혜택 구성에 따라 다르게 책정됩니다 ...",62    "바로카드 연회비는 국내 전용과 해외 겸용 여부에 따라 차등 부과됩니다 ...",63    "전월 실적 등 조건을 충족하면 다음 해 연회비가 면제되는 카드도 있습니다 ...",64    "카드 분실 신고는 고객센터 또는 앱에서 즉시 가능합니다 ...",65    ...66]67 68# Queries: inject the instruction · Documents: no instruction69q_emb = model.encode(queries, prompt=QUERY_INSTRUCTION)70d_emb = model.encode(documents)71 72scores = model.similarity(q_emb, d_emb)   # cosine; rank documents by score73print(scores)74```75 76> The instruction is also stored in the model config, so `model.encode(queries, prompt_name="query")`77> is equivalent to passing `prompt=QUERY_INSTRUCTION` explicitly. Documents use no prompt78> (`prompt_name="document"` is an empty string).79 80* **Query prompt** (instruction): `Instruct: Given a web search query, retrieve relevant passages that answer the query\nQuery: `81* **Document prompt**: none82 83## 1.3. Training Data84| Dataset | Role | Size |85|---------|------|------|86| [BCAI-Finance-Kor-Embedding-Triplet](https://huggingface.co/datasets/BCCard/BCAI-Finance-Kor-Embedding-Triplet) | Training (anchor / positive / negative) | 43,394 triplets (train) |87| [BCAI-Finance-Kor-Embedding-Pair](https://huggingface.co/datasets/BCCard/BCAI-Finance-Kor-Embedding-Pair) | Corpus pool / evaluation | 36,281 unique chunks |88 89* Sources: BC Card financial QA (BCAI) + website crawl + synthetic data (chunking + multi-query generation)90* Triplets are constructed via **hard-negative mining** over the unified corpus.91 92## 1.4. Training Procedure93| Item | Value |94|------|-------|95| Method | LoRA (PEFT) |96| LoRA | r=64, alpha=128, dropout=0.05, targets = q,k,v,o,gate,up,down_proj |97| Loss | CachedMultipleNegativesRankingLoss (in-batch negatives) |98| Batch | per-device 256 (DDP) → 511 in-batch negatives per rank |99| LR / scheduler | 5e-5 / cosine, warmup_ratio 0.1, weight_decay 0.01 |100| Epochs | 3, early stopping — best checkpoint selected by validation NDCG@10 |101| Precision | bf16, gradient checkpointing |102| Hardware | 8× NVIDIA RTX PRO 6000 Blackwell (DDP) |103 104<br>105 106# 2. Evaluation107## 2.1. Setup108* **Queries**: 1,000 (held-out test split) · **Corpus**: 36,281 unique chunks109* **Protocol**: binary-relevance information retrieval; the same evaluator used during training110* **Metrics**: NDCG@10 (primary), MRR@10, Recall@{1,10}, Accuracy@1, MAP@10111* **Models compared**: base (`Qwen3-Embedding-4B`, no fine-tuning) vs. **v4 (r64 / lr5e-5 / 3ep, released)**112 113<br>114 115## 2.2. Training116<div align="center">117  <img src="figures/evaluation-train-1-1.png" alt="Training curves - loss, learning rate, validation NDCG@10 (WandB)" >118</div>119 120Trained for 3 epochs (early-stopped) with a cosine schedule; training loss decreases steadily while validation NDCG@10 climbs early and plateaus (peak ≈ 0.695 around epoch ~1.4), and the best checkpoint is selected at the peak. Curves (loss / learning rate / validation NDCG@10) are logged to Weights & Biases.121 122<br>123 124## 2.3. In-domain Retrieval Benchmark125<div align="center">126  <img src="figures/evaluation-test-1-1.png" alt="Test-set retrieval metrics - base vs v4" >127</div>128<div align="center">129  <img src="figures/evaluation-test-1-2.png" alt="Test-set retrieval metrics comparison (per metric)" >130</div>131 132| Metric | base (Qwen3-4B) | v4 (r64/5e-5/3ep) | v4 Δ vs base |133|--------|:---:|:---:|:---:|134| **NDCG@10** | **0.6508** | **0.6906** | **+0.040 (+6.1%)** |135| MRR@10 | 0.6805 | 0.7283 | +0.048 (+7.0%) |136| Recall@10 | 0.7244 | 0.7620 | +0.038 (+5.2%) |137| Recall@1 | 0.5081 | 0.5520 | +0.044 (+8.6%) |138| Accuracy@1 | 0.5950 | 0.6480 | +0.053 (+8.9%) |139| MAP@10 | 0.6013 | 0.6410 | +0.040 (+6.6%) |140 141**v4 is the released model.** Fine-tuning lifts in-domain retrieval by **roughly +7%** over the base `Qwen3-Embedding-4B`, with the largest gains on top-rank precision (Accuracy@1, Recall@1). It also surpasses the 0.6B sibling (test NDCG@10 0.6695) by **+0.021 (+3.2%)** — a modest scale gain at ~7× the parameters, so the 0.6B remains the better pick for latency-sensitive serving.142 143### Comparison with other encoders144On the *same* in-domain test set, untuned encoders — our own `Qwen3-Embedding` base (0.6B / 4B) and public multilingual SOTA models (each run with its own native prompt format) — all fall **well below this model**: domain fine-tuning beats general-purpose scale:145 146| Model | Params | NDCG@10 | MRR@10 | Recall@10 | Accuracy@1 | MAP@10 | Avg |147|-------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|148| LiquidAI/LFM2.5-Embedding-350M | 0.35B | 0.5983 | 0.6166 | 0.6799 | 0.5320 | 0.5519 | 0.5957 |149| Qwen3-Embedding-0.6B (base) | 0.6B | 0.6186 | 0.6449 | 0.7046 | 0.5560 | 0.5652 | 0.6179 |150| google/embeddinggemma-300m | 0.3B | 0.6373 | 0.6664 | 0.7082 | 0.5790 | 0.5906 | 0.6363 |151| BAAI/bge-m3 | 0.6B | 0.6426 | 0.6660 | 0.7261 | 0.5730 | 0.5913 | 0.6398 |152| intfloat/multilingual-e5-large | 0.6B | 0.6476 | 0.6722 | 0.7313 | 0.5790 | 0.5958 | 0.6452 |153| Qwen3-Embedding-4B (base) | 4B | 0.6508 | 0.6805 | 0.7244 | 0.5950 | 0.6013 | 0.6504 |154| MoAI-Embedding-0.6B (sibling) | 0.6B | 0.6695 | 0.7060 | 0.7508 | 0.6190 | 0.6171 | 0.6725 |155| **MoAI-Embedding-4B (this model)** | 4B | **0.6906** | **0.7283** | **0.7620** | **0.6480** | **0.6410** | **0.6940** |156 157This model improves over its own `Qwen3-Embedding-4B` base by **+0.040 NDCG@10 (+6.1%)** and leads the best general-purpose baseline (e5-large) by **+0.043 NDCG@10**. Notably, the untuned **4B base (`0.6508`) trails the fine-tuned 0.6B sibling (`0.6695`)** — fine-tuning outweighs scale. _Caveat: these baselines are not tuned on BC Card data — the comparison illustrates the value of domain adaptation, not a defect in the baselines._158 159<br>160 161## 2.4. Limitations162* **Domain-specific** — tuned for BC Card Korean financial text; out-of-domain or non-Korean performance is not guaranteed.163* **Compute cost** — at 4B, this model is markedly heavier (memory / latency) than the [0.6B sibling](https://huggingface.co/BCCard/MoAI-Embedding-0.6B); for latency- or throughput-sensitive serving, consider the 0.6B variant.164* **Re-ranking recommended** — as a bi-encoder it favors recall over fine-grained precision.165    - Recommended pipeline: **Bi-Encoder (this model) Top-K → Cross-Encoder re-ranking**166* **Sequence length** — inputs are truncated at 1,024 tokens; content past that limit is not encoded, so very long documents should be chunked before indexing.167* **Exact-value matching** — fine-grained numeric/tabular facts (fees, rates, dates, terms) are not reliably distinguished by dense similarity alone; pair with lexical (BM25) retrieval or a re-ranker when exactness matters.168* **Retrieval only** — this is an embedding model, not a generator; it ranks passages and does not produce answers.169* **Synthetic data influence** — part of the training set is LLM-synthesized (chunking + multi-query), which may carry the generator's stylistic/coverage biases.170 171<br>172 173# 3. Future Work174* **Data quality improvement & re-training**175	- Human-annotation labeling176	- More rigorous hard-negative mining (iterative, mined with this model)177	- Broader/higher-quality data (incl. general financial corpora)178* **System-level**179	- Cross-Encoder re-ranker for precision180	- HyDE / dynamic instruction injection at query time181 182<br>183 184# 4. Meta Info185## 4.1. Citation186```bibtex187@misc{bccard2026moaiembedding4b,188  title        = {MoAI-Embedding-4B: A BC Card-Domain Korean Text Embedding Model},189  author       = {BC Card AX Team},190  year         = {2026},191  howpublished = {https://huggingface.co/BCCard/MoAI-Embedding-4B},192  note         = {LoRA fine-tune of Qwen3-Embedding-4B for BC Card-domain Korean retrieval}193}194```195 196## 4.2. See Also197* **0.6B sibling model**: [`BCCard/MoAI-Embedding-0.6B`](https://huggingface.co/BCCard/MoAI-Embedding-0.6B)198* **Training dataset**: [`BCCard/BCAI-Finance-Kor-Embedding-Triplet`](https://huggingface.co/datasets/BCCard/BCAI-Finance-Kor-Embedding-Triplet)199* **Corpus dataset**: [`BCCard/BCAI-Finance-Kor-Embedding-Pair`](https://huggingface.co/datasets/BCCard/BCAI-Finance-Kor-Embedding-Pair)200 201<br>202