DVSGlobal/transito-hn-retrieval-eval
Tránsito HN Retrieval Eval A small, source-verifiable benchmark for article-level retrieval in Honduran law. Given a masked excerpt from a Supreme Court ruling and the version of the Traffic Act in force on that date, can an embedding model retrieve an article the court cited? Why we built it General-purpose leaderboards help shortlist embedding models, but they cannot tell us which compact models work well on Honduran legal text. We built this benchmark while… See the full description on the dataset page: https://huggingface.co/datasets/DVSGlobal/transito-hn-retrieval-eval.
Tránsito HN Retrieval Eval
A small, source-verifiable benchmark for article-level retrieval in Honduran law.
Given a masked excerpt from a Supreme Court ruling and the version of the Traffic Act in force on that date, can an embedding model retrieve an article the court cited?
Why we built it
General-purpose leaderboards help shortlist embedding models, but they cannot tell us which compact models work well on Honduran legal text.
We built this benchmark while developing an on-device legal assistant for Honduras.
Dataset
We searched 20,074 Supreme Court rulings published between 2013 and 2025 and found 23 that explicitly cited the Traffic Act. After excluding three under documented rules, 20 cases remained. Their court-authored citations provide the ground truth, and every case links back to the official ruling.
The accompanying corpus contains all 127 articles of the Traffic Act, represented by 136 historical versions. We reconstructed its six amendments from the official gazettes so each ruling could be evaluated against the law in force on its date.
Fields
Method notes
The dataset preserves raw excerpts, including article numbers and verbatim quotations. Character-level spans allow both to be masked during evaluation.
All headline results use masked queries: cited article numbers are replaced with [N] and marked quotations are removed. Raw-query results are included only as a leakage analysis. Apart from replacing natural-person names with [PERSONA-X] placeholders, followed by manual review, the excerpts are unchanged.
We report citation Recall@k and citation MRR. These measure retrieval of articles cited by the court. Uncited articles remain unjudged rather than confirmed negatives.
Evaluation code and per-model results are available in the companion repository.
Baseline results
Eleven embedding models were evaluated using masked queries.
- Nemotron-3-Embed-8B achieved the highest MRR across all 20 cases: 0.597.
- EmbeddingGemma 308M ranked first on the 13 single-article cases: 0.605 MRR.
- The three Qwen3 Embedding models finished within 0.009 MRR of one another despite a 13× difference in size.
More than the leaderboard, we wanted to share the process: court-authored ground truth, historical versions of the law and span-based masking. This is still an early benchmark, but we think the results are already interesting enough to be useful and to start a broader conversation.
Collaborate
Model authors are welcome to contribute reproducible baselines. Open a GitHub issue with the exact model revision and its recommended query and document prefixes.
Built in San Pedro Sula, Honduras by DVSGlobal Lab applied AI from Central American.