Waqf-AI/arabic-splade-efficient

Arabic SPLADE — Phase 3

Efficient symmetric SPLADE using DistilBERT multilingual for faster inference.

Architecture

Symmetric shared (MLMTransformer+SpladePooling, sequential)

Base model: distilbert-base-multilingual-cased

Training

• —Dataset: oddadmix/arabic-triplets-large (104K triplets, 92K unique passages)
• —Loss: SpladeLoss(SparseMultipleNegativesRankingLoss, q_reg=5e-5, d_reg=3e-5)
• —Batch: 16 per GPU, grad accum 4
• —Learning rate: 2e-5
• —Epochs: 1
• —AMP: fp16
• —Sampler: NO_DUPLICATES

Evaluation on Arabic NanoBEIR (13 datasets)

For reference: BM25 scores 0.3824 NDCG@10, 0.4483 MRR@10 on the same benchmark.

Training Details

DistilBERT multilingual (6-layer, 119K vocab), ~2x faster than AraBERT

Hardware

• —2× NVIDIA TITAN RTX (23.5 GB each)
• —DDP via torchrun

Usage

from sentence_transformers.sparse_encoder import SparseEncoder

model = SparseEncoder("Abdelkareem/arabic-splade-efficient")
embeddings = model.encode([
    "ما هي عاصمة مصر؟",
    "القاهرة هي عاصمة مصر وأكبر مدنها.",
])
print(embeddings.shape)
# Decode top tokens
decoded = model.decode(embeddings, top_k=10)
for d in decoded:
    print(d)