Datasetpublic

neoai-inc/LIT-RAGBench

LIT-RAGBench LIT-RAGBench is a benchmark for evaluating generator capabilities in Retrieval-Augmented Generation (RAG). It focuses on whether a model can answer questions correctly given retrieved documents, independent of retrieval quality. The benchmark covers five categories: Integration, Reasoning, Logic, Table, and Abstention. Dataset Summary LIT-RAGBench contains: 114 human-constructed Japanese questions An English version generated by machine translation… See the full description on the dataset page: https://huggingface.co/datasets/neoai-inc/LIT-RAGBench.

sourceHugging Facecc-by-sa-4.0updated 6mo agoView on Hugging Face

0likes101downloads

Dataset Card

LIT-RAGBench

LIT-RAGBench is a benchmark for evaluating generator capabilities in Retrieval-Augmented Generation (RAG). It focuses on whether a model can answer questions correctly given retrieved documents, independent of retrieval quality. The benchmark covers five categories: Integration, Reasoning, Logic, Table, and Abstention.

[image]

Dataset Summary

LIT-RAGBench contains:

—114 human-constructed Japanese questions
—An English version generated by machine translation with human curation

Each example includes the following fields:

—question
—answer: reference answer
—qa_type: question type
—positive_chunk_list: relevant evidence chunks
—negative_chunk_list: irrelevant chunks
—reasoning_content: reasoning process for deriving the answer

Intended Use

This dataset is intended for:

—benchmarking LLMs used as RAG generators
—analyzing model strengths and weaknesses by category
—developing and evaluating RAG-specialized models :contentReference[oaicite:6]{index=6}

Repository

Code, prompts, and the original dataset files are available in the GitHub repository: Koki-Itai/LIT-RAGBench

Citation

bibtex

@misc{litragbench,
      title={LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation},
      author={Koki Itai and Shunichi Hasegawa and Yuta Yamamoto and Gouki Minegishi and Masaki Otsuki},
      year={2026},
      eprint={2603.06198},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2603.06198},
}

License

The code and data are released under the Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license. Commercial use is permitted provided that you follow the license terms, including attribution and sharing any derivative works under the same.

[![CC BY-SA 4.0][cc-by-sa-image]][cc-by-sa]

[cc-by-sa]: http://creativecommons.org/licenses/by-sa/4.0/ [cc-by-sa-image]: https://licensebuttons.net/l/by-sa/4.0/88x31.png [cc-by-sa-shield]: https://img.shields.io/badge/License-CC%20BY--SA%204.0-lightgrey.svg