Boakpe/environmental_registry_test_set

Environmental Registry Test Set

This dataset is the anonymized primary benchmark used for evaluating agentic Portuguese Text-to-SQL over a real PostgreSQL/PostGIS environmental-registry database. The underlying production database is not released, but the benchmark metadata and gold labels are provided for transparency and comparison.

Code and reproducibility repository:

https://github.com/Boakpe/distilled-slms-for-text-to-sql-pt-br

Related collection:

https://huggingface.co/collections/Boakpe/distilled-slms-for-text-to-sql-pt-br

Dataset Summary

• —Rows: 180
• —Split: test
• —SQL questions: 90
• —Clarification questions: 45
• —Unanswerable questions: 45
• —SQL difficulty split: 20 easy, 30 medium, 30 hard, 10 expert
• —Language: Brazilian Portuguese
• —Database type: PostgreSQL/PostGIS environmental-registry database

The primary schema contains 19 tables and covers rural properties, municipalities, Indigenous Lands, conservation units, environmental embargoes, settlements, quilombola territories, federal public lands, deforestation events, and spatial-overlap facts.

Columns

• —id: benchmark identifier.
• —pergunta: natural-language question in Portuguese.
• —sql: gold SQL for SQL questions, or terminal label such as clarification / unanswerable for non-SQL questions.
• —dificuldade: SQL difficulty label.
• —categorias: topic labels.
• —order_matters: whether row order is semantically required.
• —num_column_matters: whether exact column count is required.
• —num_row_matters: whether exact row count is required.

Evaluation Design

The benchmark combines:

• —SQL generation.
• —Ambiguity detection through clarification questions.
• —Unanswerability detection.
• —PostGIS/geospatial reasoning.
• —Temporal filters, joins, aggregations, and domain-specific value grounding.

SQL evaluation uses execution comparison rather than exact SQL-string match. The reported primary metric is relaxed execution accuracy, with strict execution accuracy also reported as a diagnostic.

Reported Results

Primary environmental-registry benchmark, Pass@1:

Fine-tuned model Pass@K:

License

Apache 2.0.