VISAI-AI/nitibench

👩🏻‍⚖️ NitiBench: A Thai Legal Benchmark for RAG

[[📄 Technical Report](https://arxiv.org/pdf/2502.10868)] | [[👨‍💻 Github Repository](https://github.com/vistec-AI/nitibench/)]

This dataset provides the test data for evaluating LLM frameworks, such as RAG or LCLM. The benchmark consists of two datasets:

• —NitiBench-CCL
• —NitiBench-Tax

🏛️ NitiBench-CCL

Derived from the WangchanX-Legal-ThaiCCL-RAG Dataset, our version includes an additional preprocessing step in which we separate the reasoning process from the final answer. The dataset contains 35 pieces of legislation related to Corporate and Commercial Law (CCL). Information about the 35 pieces of legislation is provided in the table below:

The training split of nitibench-ccl can be found in the WangchanX-Legal-ThaiCCL-RAG dataset.

Data Format

Each data point contains four columns:

• —question: str — A question relevant to the relevant_laws.
• —answer: str — The original answer generated by an LLM, which has been revised and edited by legal experts to include both the reasoning steps and the final answer.
• —relevant_laws: List[Dict[str, str]] — A list of relevant law name, section, and contents.
• —reference_answer: str — The answer to the question based on the relevant_laws, provided without the reasoning steps.
• —reference_laws: List[Dict[str, str]] - A list of referenced law mentioned in relevant_laws column.

Formally, given the data triple \((q, T=\{p1, p2, \dots, pK\}, y)\), \(q\) represents the `question`, \(T\) represents `relevantlaws, and \(y\) represents the answer`.

Data Curation

Using the notation described above, the data was curated as follows:

1. 1.Queries (\(q\)) and answers (\(y\)) were manually crafted by legal experts based on a single section sampled from the legal texts of the 35 pieces of legislation.
2. 2.For each data triple \((q, T, y)\), the manually crafted question was carefully quality-assured by a second legal expert.

Thus, for the test data, there is only one positive per query (\(|T|=1\)). The diagram below shows how the test data was collected.

[image]

💸 NitiBench-Tax

This subset provides a question, relevant laws, and an answer for each data point. Instead of having legal experts manually craft the questions, we scraped the data from a reliable source: the Revenue Department Website. This subset contains Tax Ruling Cases officially provided by the Revenue Department since 2021. As a result, this subset is considerably more challenging, as it requires extensive legal reasoning both for searching for relevant documents and for generating the answer. The data collection procedure is illustrated in the figure below:

[image]

Data Format

This split uses the same format as described in the NitiBench-CCL split.

Contact

For any inquiries or concerns, please reach out to us via email: Chompakorn Chaksangchaichot.

Citation

@inproceedings{akarajaradwong-etal-2025-nitibench,
    title = "{N}iti{B}ench: Benchmarking {LLM} Frameworks on {T}hai Legal Question Answering Capabilities",
    author = "Akarajaradwong, Pawitsapak  and
      Pothavorn, Pirat  and
      Chaksangchaichot, Chompakorn  and
      Tasawong, Panuthep  and
      Nopparatbundit, Thitiwat  and
      Pratai, Keerakiat  and
      Nutanong, Sarana",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1739/",
    doi = "10.18653/v1/2025.emnlp-main.1739",
    pages = "34292--34315",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language models (LLMs) show promise in legal question answering (QA), yet Thai legal QA systems face challenges due to limited data and complex legal structures. We introduce NitiBench, a novel benchmark featuring two datasets: (1) NitiBench-CCL, covering Thai financial laws, and (2) NitiBench-Tax, containing Thailand{'}s official tax rulings. Our benchmark also consists of specialized evaluation metrics suited for Thai legal QA. We evaluate retrieval-augmented generation (RAG) and long-context LLM (LCLM) approaches across three key dimensions: (1) the benefits of domain-specific techniques like hierarchy-aware chunking and cross-referencing, (2) comparative performance of RAG components, e.g., retrievers and LLMs, and (3) the potential of long-context LLMs to replace traditional RAG systems. Our results reveal that domain-specific components slightly improve over naive methods. At the same time, existing retrieval models still struggle with complex legal queries, and long-context LLMs have limitations in consistent legal reasoning. Our study highlights current limitations in Thai legal NLP and lays a foundation for future research in this emerging domain."
}

@misc{akarajaradwong2025nitibenchcomprehensivestudiesllm,
      title={NitiBench: A Comprehensive Studies of LLM Frameworks Capabilities for Thai Legal Question Answering}, 
      author={Pawitsapak Akarajaradwong and Pirat Pothavorn and Chompakorn Chaksangchaichot and Panuthep Tasawong and Thitiwat Nopparatbundit and Sarana Nutanong},
      year={2025},
      eprint={2502.10868},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.10868}, 
}

License

This dataset is provided under the MIT License.

Acknowledgment

We sincerely appreciate the generous support from the WangchanX program sponsors—PTT, SCB, and SCBX—whose funding made this project possible. We are also grateful for the invaluable collaboration with VISTEC, which was crucial in bringing this project to fruition.

<br> <div style="display: flex; align-items: center; gap: 20px;"> Sponsored by <img src="./assets/VISAILogoHorizontal.png" style="height:30px;" alt="VISAI Logo"> <img src="./assets/Logo_vistec.png" style="height:30px;" alt="VISTEC Logo"> </div>