thivy/eti-embedding-training-data-2048-triplets-v4

ETI Embedding Training Data — v4 (5-style, LLM-judged triplets)

Norwegian (nb) retrieval triplets built from the `NorskHelsenett/LOS_Document_classification_ETI` corpus of public-service / welfare / health documents.

Each row is a triplet (anchor, positive, negative) plus three metadata columns (style, category, doc_url) you can use to filter, weight, or build curriculum stages.

76,408 triplets · 2,530 source documents · 38,629 distinct anchors.

Schema

Construction pipeline

Built by a five-stage pipeline run on Azure OpenAI:

Stage A — anchor generation (per chunk)
  └─ 5 styles × every chunk, generated by gpt-4.1
     tight_canonical   "Hjelpestønad ved langvarig sykdom"     ← rewriter-output-shaped
     loose_canonical   "Hva er hjelpestønad?"                  ← canonical noun + light scaffolding
     vague_metaphor    "Litt drahjelp når man står helt fast"  ← metaphor, no canonical noun
     mid_parent        "Vi sliter med å få endene til å møtes, finnes det noe vi kan søke?"
     long_parent       (3–5 sentence parent narrative)
Stage B — DOMAIN VOCAB paraphrase pairs (everyday ↔ canonical Norwegian welfare terms)
Stage C — Antipattern hard-negative descriptions (synthesised failure modes of the rewriter prompt)
Stage D — FAISS hard-negative mining (anchor → top-k chunks from a different doc_url)
Stage E — LLM-as-judge filter
  └─ gpt-5.1 evaluates every (anchor, positive, negative) — drops triplets where
     the model judges the negative more relevant than the positive (false negatives).
     ~87% of mined triplets survived.

Same-document chunks are excluded from the negative pool to prevent near-duplicate trivial pairs.

Why so many styles?

The target deployment is a two-stage RAG pipeline:

parent query  →  rewriter LLM  →  embedding model  →  retrieved docs

The rewriter prompt is itself under iterative development — its exact output distribution will shift over time. Training the embedding model on multiple anchor styles spanning the full pipeline (raw parent language → canonical rewriter output → metaphor) makes the embedding robust to those shifts, instead of overfitting to today's rewriter behaviour.

How v4 differs from v3

`thivy/eti-embedding-training-data-2048-triplets-v3` was a single-purpose embedding-training set. v4 is a superset of v3's ideas designed to also work for reranker training and evaluation.

Recommended uses

1. Bi-encoder / embedding fine-tuning

Drop-in for CachedMultipleNegativesRankingLoss or TripletLoss in sentence-transformers. The five style buckets give the model a wide distribution of anchor surfaces; the same-doc filter on negatives prevents trivial-pair overfitting.

from sentence_transformers import SentenceTransformer, losses
from datasets import load_dataset

ds = load_dataset("thivy/eti-embedding-training-data-2048-triplets-v4", split="train")
ds = ds.select_columns(["anchor", "positive", "negative"])
loss = losses.CachedMultipleNegativesRankingLoss(model, mini_batch_size=24)

2. Cross-encoder / reranker training

Stage E (gpt-5.1 judge) gives every triplet a verified relevance ordering — positive > negative is not a heuristic, it's a strong-LLM verdict. That makes v4 well-suited for cross-encoder training, where clean pairwise labels matter more than volume.

Treat each row as two labelled pairs:

# pseudo-code
for row in ds:
    yield (row["anchor"], row["positive"]), 1.0
    yield (row["anchor"], row["negative"]), 0.0

For curriculum learning, increase difficulty by sorting on style: tight_canonical → loose_canonical → mid_parent → long_parent → vague_metaphor.

3. Retrieval evaluation

Use doc_url to build a clean held-out eval set without document leakage:

held_out_docs = {url for i, url in enumerate(ds["doc_url"]) if hash(url) % 10 == 0}
eval_set  = ds.filter(lambda x: x["doc_url"] in held_out_docs)
train_set = ds.filter(lambda x: x["doc_url"] not in held_out_docs)

Style distribution

Source data

Derived from `NorskHelsenett/LOS_Document_classification_ETI` — a Norwegian public-service corpus covering welfare, health, education, work and culture services. Chunks were produced with LlamaIndex SemanticSplitterNodeParser (fallback SentenceSplitter at 2,048 tokens).

Caveats

• —All anchors are LLM-generated; they are stylistically diverse but they are not real user queries. Pair with a small real-query eval set before claiming production gains.
• —Judge filtering relies on gpt-5.1; residual judge errors are inherited by the dataset.
• —The vocab_* styles are rare (~0.8% combined). Up-weight them if you specifically want to bake in the welfare-vocabulary mappings.
• —Hard negatives are FAISS-mined from a single encoder snapshot; a bootstrapped second round of mining against the trained model would likely surface more discriminative negatives.

Citation

@misc{eti-embedding-training-data-v4,
  author       = {thivy},
  title        = {ETI Embedding Training Data — v4 (5-style, LLM-judged)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/thivy/eti-embedding-training-data-2048-triplets-v4}}
}