EchoesML/bge-m3-georgian-literary

BGE-M3 Georgian — Fine-tuned Embedding Model

A fine-tuned version of BAAI/bge-m3 for dense retrieval over Georgian literary text. Trained on 14,340 triplets derived from 9 classical Georgian authors (2,390 chunks, sourced from ka.wikisource.org).

Benchmark Results

Evaluated on a held-out test set of 7,170 queries against 2,390 passages — all unseen during training.

Fine-tuning improved Recall@8 from 0.517 → 0.7278 (+40.8% relative gain over the base model).

Intriguing Questions

The full methodology is documented in two companion files. But here are some questions that might pull you in:

→ Why does the averaged checkpoint outperform the best single checkpoint — almost every time? Each epoch's model lands in a slightly different region of the loss landscape. Individual checkpoints sit in sharp minima — well-fitted to their specific batch order, but brittle. The averaged model lands in the center of the region explored by the last three epochs: a flatter, wider basin that generalises better to queries the model has never seen. No additional training. Free improvement. [Full explanation with loss landscape intuition → training.md]

→ Why did I abandon the most popular chunking strategy (sliding window with overlap) — and get better results with a simpler one? Sliding window overlap is widespread because it helps boundary sentences appear in multiple chunks, raising their retrieval odds. But it comes at a cost: your index contains duplicated content, reconstruction score drops below 1.0, and you lose the ability to map any retrieved chunk back to its exact position in the source. The sentence-boundary nearest-neighbor strategy used here hits a 1.0 reconstruction score on all 28 source files, keeps every chunk within tight size bounds (500–1200 chars), and never embeds the same character twice. [Full algorithm with edge cases → chunking_strategy.md]

→ How did a smaller model with worse tokenization beat a larger model with a purpose-built tokenizer? Qwen3-Embedding-0.6B (596M parameters) tokenizes Georgian whole-word — it sees ვეფხისტყაოსანი as a single token, not fragmented subwords. BGE-M3 (568M parameters, XLM-RoBERTa based) chops the same word into pieces. By every surface measure, Qwen3 should win on Georgian. It didn't — not even close (Recall@8: 0.2685 vs 0.7278). The reason is pretraining: BGE-M3 was pretrained on 1.2 billion retrieval pairs across 100+ languages. Its representations are already calibrated for semantic similarity at the vector level. Qwen3 Embedding's superior tokenization raises the ceiling of what fine-tuning can achieve, but the floor it started from was so much lower that three epochs of LoRA couldn't close the gap. Tokenizer quality is not the same as retrieval quality. [Full breakdown with training configs → training.md]

Model Details

Training Data

Companion Documentation

• —`training.md` — Full training methodology: data pipeline, hard negative mining, why the custom training loop caused catastrophic forgetting, checkpoint averaging explained with loss landscape geometry.
• —`chunking_strategy.md` — The chunking algorithm in full: sentence-boundary search, reconstruction scoring, why overlap was rejected, contextual chunking with author/title metadata, and the script annotated line by line.

License

Model weights: Apache 2.0 (inherited from BAAI/bge-m3). Training corpus: public domain (ka.wikisource.org).