danielnoumon/qwen3-embedding-4b-ai-act-nl

Qwen3-Embedding-4B EU AI Act NL

Fine-tuned Qwen3-Embedding-4B with LoRA for Dutch/English retrieval on EU AI Act documentation. Supports Matryoshka embeddings (2560, 1024, 768, 512, 256, 128 dimensions) for flexible speed/quality tradeoffs.

Model Details

• —Base model: Qwen/Qwen3-Embedding-4B
• —Architecture: Decoder-based (Qwen3), last-token pooling, left padding
• —Training approach: Two-stage fine-tuning with LoRA (r=16, alpha=32)
• —Stage 1: CachedMNRL + Matryoshka on synthetic query-chunk pairs
• —Stage 2: CachedMNRL + Matryoshka with hard negatives mined from Stage 1 model
• —Dataset: 1,944 synthetic queries generated from EU AI Act chunks (Dutch/English)
• —Hardware: NVIDIA RTX 5090 (32GB VRAM, Blackwell)
• —Precision: bf16 + SDPA

Performance

Evaluated on 340 held-out queries across 85 chunks. All metrics measured with cosine similarity.

NDCG@10 across Matryoshka dimensions

Stage 2 results are essentially equivalent to Stage 1 — the model was already near-ceiling after Stage 1 on this dataset.

Full metrics at dim=2560

Full metrics at dim=1024

Usage

Installation

pip install sentence-transformers>=2.7.0 transformers>=4.51.0

Basic usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("danielnoumon/qwen3-embedding-4b-ai-act-nl")

# Qwen3 uses instruct prompts for queries, no prefix for documents
queries = model.encode(
    ["What are the obligations for high-risk AI systems?"],
    prompt="Instruct: Given a question about EU AI regulation, retrieve the most relevant passage\nQuery:",
)
passages = model.encode([
    "High-risk AI systems must comply with requirements in Chapter III...",
    "The AI Act defines prohibited practices in Article 5...",
])

# Compute similarity
from sentence_transformers.util import cos_sim
scores = cos_sim(queries, passages)

Matryoshka embeddings (dimension truncation)

# Encode with full 2560 dimensions
embeddings_2560 = model.encode(queries)

# Truncate to 256 dimensions for faster search
embeddings_256 = embeddings_2560[:, :256]

# Or specify dimension at encoding time
model.truncate_dim = 256
embeddings_256 = model.encode(queries)

Speed vs quality tradeoff:

• —dim=2560: Full quality (NDCG@10 = 0.962)
• —dim=1024: Marginally better subspace (NDCG@10 = 0.966) — Matryoshka effect
• —dim=256: ~90% fewer dimensions, 97.9% of dim=1024 quality (NDCG@10 = 0.942)
• —dim=128: ~95% fewer dimensions, 95.4% of dim=1024 quality (NDCG@10 = 0.919)

Important: Use instruct prompts

Qwen3 uses instruction-based prompting. Queries need the instruct prefix, documents do not:

# Queries: use instruct prompt
query_emb = model.encode(
    ["your question here"],
    prompt="Instruct: Given a question about EU AI regulation, retrieve the most relevant passage\nQuery:",
)

# Documents: no prefix needed
doc_emb = model.encode(["your document here"])

Training Details

LoRA Configuration

Stage 1: CachedMNRL + Matryoshka

• —Loss: MatryoshkaLoss(CachedMultipleNegativesRankingLoss)
• —Matryoshka dims: [2560, 1024, 768, 512, 256, 128]
• —Batch size: 128 (GradCache), mini-batch 2
• —Learning rate: 1e-4 (typical for LoRA)
• —Epochs: 3
• —Negatives: 127 in-batch negatives per query (via GradCache)
• —Precision: bf16 + SDPA

Stage 2: Hard negatives

• —Starting point: Stage 1 LoRA checkpoint (merged into base)
• —Hard negative mining: Top-1 most similar wrong chunk per query (using Stage 1 model)
• —Learning rate: 1e-5 (10× lower to prevent catastrophic forgetting)
• —Epochs: 2
• —Batch size: 128 (GradCache), mini-batch 1
• —Negatives: 1 explicit hard negative + 127 in-batch negatives

Dataset

• —Dataset: danielnoumon/eu-ai-act-nl-queries
• —Train: 1,944 synthetic query-chunk pairs
• —Eval: 340 queries × 85 chunks
• —Split strategy: Chunk-level (no chunk appears in both train and eval)
• —Query generation: Azure OpenAI GPT-4o-mini with structured prompts

Hardware Notes

• —bf16 works on Blackwell (RTX 5090) with Qwen3
• —Qwen3's RMSNorm upcasts to fp32 internally
• —CachedMNRL (GradCache) essential for fitting large contrastive pools in 32GB VRAM with a 4B model
• —LoRA keeps trainable params at 0.29% — full fine-tuning would exceed 32GB VRAM
• —flashattention2 recommended but not required (sdpa works as fallback)

Limitations

• —Domain-specific: Fine-tuned on EU AI Act documentation. Performance on other domains may vary.
• —Language: Optimized for Dutch and English. Other languages supported by the base model may work but are not evaluated.
• —Chunk size: Trained on chunks up to 512 tokens. Very long documents should be chunked.

License

Apache 2.0

Citation

@misc{qwen3embedding,
    title={Qwen3-Embedding: Advancing Text Embeddings with Qwen3},
    author={Qwen Team},
    year={2025},
    url={https://huggingface.co/Qwen/Qwen3-Embedding-4B}
}

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    year = "2019",
    url = "https://arxiv.org/abs/1908.10084",
}

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

@misc{hu2022lora,
    title={LoRA: Low-Rank Adaptation of Large Language Models},
    author={Edward J. Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},
    year={2022},
    eprint={2106.09685},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}