gvic-unb/dodf-saude-qa

DODF Public Health QA

Dataset Summary

DODF Public Health QA is a synthetic question-answering (QA) dataset for evaluating Retrieval-Augmented Generation (RAG) systems over official public health publications from the Diário Oficial do Distrito Federal (DODF), the Government Gazette of the Federal District, Brazil.

The dataset focuses on location-aware questions about public health facilities and services — such as Basic Health Units (UBS), Emergency Care Units (UPA), hospitals, and health programs — distributed across the 58 administrative regions of the Federal District.

Dataset Structure

The dataset contains three splits:

QA splits (test and validation)

Each instance contains the following fields:

Knowledge base split (corpus)

Each instance corresponds to a single DODF publication and contains the following fields:

Knowledge Base Construction

The knowledge base comprises 1,381 publications drawn from 229 editions of the DODF issued in 2025. Publications were prioritized from Sections I and III, which cover government acts, administrative decisions, contracts, and public health policies. Section II, which covers internal personnel matters, was excluded. Publications were filtered for public health relevance using rule-based criteria, retaining only documents mentioning health facilities, programs, or services pertaining to the 58 administrative regions of the Federal District.

QA Corpus Construction

While human-authored questions remain the gold standard, producing them is a demanding task that requires experts to review a large volume of publications. Consequently, generating synthetic test sets has become a widely adopted alternative, wherein an LLM produces questions from reference documents. Following this approach, QA pairs were generated synthetically from the knowledge base using Claude Haiku 4.5 (claude-haiku-4-5-20251001), with prompts designed with the support of a public health expert to simulate the inquiry style of researchers, managers, and citizens.

Generation

Two types of location-aware questions were produced:

• —Type I — questions whose answers are found within a single DODF publication.
• —Type II — questions that require information aggregated from multiple publications (up to 5 documents) for a specific location. For these, all publications mentioning a given administrative region (e.g., Ceilândia) are grouped together, and the model is prompted to generate questions whose answers span the group's content.

Filtering

Generated pairs were filtered through the following steps:

1. 1.Pairs with empty question or answer fields were discarded.
2. 2.Pairs whose questions did not contain any target location from the reference dataset were removed via regex matching.
3. 3.Pairs whose questions showed no lexical relation to public health terminology (e.g., saúde, UBS, hospital, SUS) were discarded through a keyword-based heuristic.
4. 4.A lexical grounding check verified that at least 30% of the answer tokens appear in the source text, mitigating hallucinated responses.

Statistics

After filtering, pairs were shuffled and split: 150 samples were selected for the validation set (used to refine prompts and tune LLM hyperparameters such as temperature and top-K), and 350 samples were kept for the final evaluation set.

Languages

Brazilian Portuguese (pt-BR).

Licensing

This corpus is released under the Creative Commons Attribution 4.0 International (CC BY 4.0) license.

Citation

If you use this corpus, please cite:

@inproceedings{freire2026locationaware,
  title = {Evaluating Location-Aware {RAG} Approaches for Question Answering over Public Health--Related Content in {Brazilian} Official Gazettes},
  author = {Freire, Arthur S. M. and Borges, Vinicius R. P. and Martins, Anna Carolina F. and Vidal, Fl{\'a}vio de Barros and Araujo, Aleteia and
            Briga{\~{a}}o, Fabr{\'i}cio R. and Veloso, Gabriel M. and Maia, Edward T. and Martins, Wagner de Jesus and Garcia, Luis P. F.},
  booktitle = {36th Brazilian Conference on Intelligent Systems, {BRACIS} 2026, Cuiabá, Mato Grosso, October 19-22 (to appear)},
  publisher = {Springer},
  year = {2026}
}

Contact

For questions or feedback, please open an issue in the project repository.