Mollel/hazina-fiscal-finetune

Hazina Fiscal Embedding Fine-tune (Swahili–English)

A self-supervised contrastive training set for fine-tuning sentence-embedding / retriever models on Tanzanian fiscal & public-finance text. Built from the same 10 Government of Tanzania documents as the `hazina-fiscal-retrieval` evaluation benchmark, but in (anchor, positive, hard-negatives) form ready for MultipleNegativesRankingLoss.

Target base model: `intfloat/multilingual-e5-base` (works with any multilingual encoder; E5 prefixes are applied at training time, not stored in the data).

Format

from datasets import load_dataset
ds = load_dataset("Mollel/hazina-fiscal-finetune", split="train")
ds[0].keys()  # anchor, positive, negative_1..negative_5

train_pairs_with_metadata.jsonl (in the repo) carries the same pairs plus strategy, lang, and positive_id for inspection; it is not the training file.

How the pairs were built (no LLM used)

Two classic self-supervised retrieval signals, then BM25 hard negatives:

1. 1.Inverse Cloze Task (ICT) — a sentence is pulled out as the anchor; the remaining passage becomes the positive (the sentence is kept 10% of the time, à la ORQA/REALM). This is the asymmetric query→passage signal. (~3.1k pairs)
2. 2.Independent cropping (Contriever-style) — two non-overlapping spans of the same passage form a positive pair. (~1.2k pairs)
3. 3.Hard negatives — for each anchor, the top BM25 passages are taken as negatives, excluding the positive and any chunk on an adjacent page of the same document (to avoid false negatives), and matched to the positive's language.

Quality filters drop number-soup table rows, acronym/abbreviation glossaries, and non-prose fragments (digit-ratio, stopword-density, and acronym-ratio gates).

Train it

pip install -U sentence-transformers datasets accelerate
python scripts/train_e5.py \
    --model intfloat/multilingual-e5-base \
    --train train.parquet \
    --eval_dir ../hazina-fiscal-retrieval \
    --epochs 2 --batch 32

The script applies E5's query: / passage: prefixes, trains with MultipleNegativesRankingLoss (+ NO_DUPLICATES batch sampler), and evaluates each epoch on the held-out hazina-fiscal-retrieval benchmark with an InformationRetrievalEvaluator (nDCG@10, Recall@10, MRR@10). Pass --prompt none for prefix-free models such as BAAI/bge-m3.

Train / eval separation

Training pairs share the corpus with hazina-fiscal-retrieval (expected for domain adaptation) but use synthetic anchors that are disjoint from the 55 hand-authored eval queries. So the benchmark remains a fair held-out test: the BM25 baseline there was nDCG@10 ≈ 0.18 on cross-lingual queries — the headroom this set is meant to close.

Limitations

• —Self-supervised anchors are passage-derived, not natural user questions, and are language-preserving (no sw↔en pairs). To add natural-language and cross-lingual signal, generate questions per chunk with an LLM and merge them in as extra (anchor, positive) rows.
• —BM25 hard negatives can occasionally be true positives (residual false-negative risk); the adjacent-page exclusion mitigates but does not eliminate this.

Attribution

Derived from publicly distributed Government of Tanzania publications (Ministry of Finance / National Audit Office). For non-commercial research; verify terms before commercial use.