VISAI-AI/thai-gazette-evidence-retrieval

Thai Royal Gazette Evidence Retrieval

A retrieval dataset over notices from the Thai Royal Gazette (ราชกิจจานุเบกษา ratchakitcha 'Royal Gazette'). Each question links to the documents that answer it. Each link gives the exact character span of the evidence in the document. The dataset contains questions, documents, and relevance judgments. It contains no generated answers.

Layout

The dataset uses the MTEB retrieval layout of one repository and three configs. Standard BEIR and MTEB loaders work without changes.

from datasets import load_dataset

corpus = load_dataset("VISAI-AI/thai-gazette-evidence-retrieval", "corpus", split="corpus")
queries = load_dataset("VISAI-AI/thai-gazette-evidence-retrieval", "queries", split="queries")
qrels = load_dataset("VISAI-AI/thai-gazette-evidence-retrieval", split="test")  # default config

corpus

_id is the Gazette notice identifier. text is the OCR plaintext of the notice. title is empty for every document. It exists only because BEIR-style loaders expect the column. category is the notice type, for example bankruptcy.

queries

question_type is single_source for the 1,500 questions that one notice answers. It is multi_source for the 500 questions that need two notices. Some questions cite evidence that resolves to one notice. An example is an amendment that quotes both the original text and the changed text. These questions carry the single_source label. author_model names the model that wrote each question. judge_model names the model that accepted it.

default (qrels)

The table has one row for each question, document, and evidence span. score is always 1. A document is either required evidence or absent. A multi-source question has one row for each document it needs.

The dataset gives evidence at two levels. This follows the minimal-answer and passage distinction of TyDi QA and Natural Questions.

• —evidence_text is the exact quote that supports the answer. It is the narrowest target for span-level scoring.
• —evidence_start_char and evidence_end_char locate that quote in the document. They are half-open character offsets, which are Python string indices and not bytes. Every row satisfies corpus[corpus-id].text[start:end] == evidence_text.
• —passage_text is the paragraph that contains the quote, with whitespace stripped. It always contains evidence_text. It gives enough context to check an answer without reading the full notice.

You can score a retriever at document level with query-id and corpus-id alone. You can also score it at span level against the character offsets.

Construction

Documents come from OCR of Royal Gazette PDFs. They contain OCR errors. A language model read a seed notice and wrote each question. A second model then screened each question against four criteria.

1. 1.The question stands alone and does not depend on context.
2. 2.The cited span supports the stated answer.
3. 3.A person would plausibly ask the question.
4. 4.A multi-source question needs every document it cites.

The pipeline discarded every question that failed a check.

Limitations

Models wrote and screened the questions with human review only convering a sample. Domain experts did not write them, and people searching the Gazette for their own reasons did not write them. The questions are therefore a proxy for real information need.

The corpus contains OCR noise, and so do some evidence spans. A question can rest on a date or a name that OCR invented. The screening pass read the source excerpts and rejected the clearest of these, but it might not have removed all of them.

License

Section 7 of the Copyright Act B.E. 2537 excludes notices of Thai government agencies from copyright protection. The questions, relevance judgments, and span annotations in this dataset use the CC BY 4.0 license.