Boakpe/rede_saude_publica_test_set

Rede Saude Publica Test Set

This dataset is the public-health transfer benchmark for the released Text-to-SQL agent artifact. It is a synthetic Brazilian public-health schema and test set used to measure cross-database generalization: the fine-tuned model was not trained on trajectories from this schema.

Code and reproducibility repository:

https://github.com/Boakpe/distilled-slms-for-text-to-sql-pt-br

Related collection:

https://huggingface.co/collections/Boakpe/distilled-slms-for-text-to-sql-pt-br

Dataset Summary

• —Rows: 100
• —Split: test
• —SQL questions: 50
• —Clarification questions: 25
• —Unanswerable questions: 25
• —SQL difficulty split: 16 easy, 21 medium, 10 hard, 3 expert
• —Language: Brazilian Portuguese
• —Database domain: synthetic Brazilian public-health network

The database covers health units, professionals, patients, appointments, hospitalizations, bed occupancy, medicine stock, and vaccination campaigns.

Columns

• —id: benchmark identifier.
• —pergunta: natural-language question in Portuguese.
• —sql: gold SQL for SQL questions, or terminal label such as clarification / unanswerable for non-SQL questions.
• —dificuldade: SQL difficulty label.
• —categorias: topic labels.
• —order_matters: whether row order is semantically required.
• —num_column_matters: whether exact column count is required.
• —num_row_matters: whether exact row count is required.

Evaluation

The benchmark is designed for the agent runtime in the GitHub repository:

https://github.com/Boakpe/distilled-slms-for-text-to-sql-pt-br

Runnable database artifact:

cd artifact/rede_saude_publica_agent
uv sync
docker compose up -d
uv run python -m agent.cli.init_db --database rede_saude_publica --recreate
uv run python -m agent.cli.eval_sql_benchmark \
  --database rede_saude_publica \
  --eval-scope all \
  --model 0 \
  --agent-verbose 1

SQL rows are evaluated by executing generated and reference SQL against the same PostgreSQL database. Non-SQL rows are evaluated by terminal action type.

Reported Results

Out-of-domain rede_saude_publica benchmark:

Intended Use

• —Cross-database evaluation for Portuguese Text-to-SQL agents.
• —SQL, clarification, and unanswerability evaluation.
• —Reproducible local benchmarking with PostgreSQL.

This is a synthetic benchmark and should not be interpreted as real medical data.

License

Apache 2.0.