CoolFace
Datasetpublic

jinulee-v/expert-rag-benchmarks

Expert RAG Benchmarks A unified collection of four expert-level legal RAG benchmarks, exposed as six named splits and three relational configurations: questions, documents, and qrels. The KCL split is named kcl_essay because Hugging Face split identifiers do not permit hyphens; its source name remains kcl-essay. Loading from datasets import load_dataset repo_id = "jinulee-v/expert-rag-benchmarks" questions = load_dataset(repo_id, "questions", split="housing")… See the full description on the dataset page: https://huggingface.co/datasets/jinulee-v/expert-rag-benchmarks.

sourceHugging Faceotherupdated 3d agoView on Hugging Face
0likes625downloads
Dataset Card

Expert RAG Benchmarks

A unified collection of four expert-level legal RAG benchmarks, exposed as six named splits and three relational configurations: questions, documents, and qrels.

The KCL split is named kcl_essay because Hugging Face split identifiers do not permit hyphens; its source name remains kcl-essay.

Loading

python
from datasets import load_dataset

repo_id = "jinulee-v/expert-rag-benchmarks"
questions = load_dataset(repo_id, "questions", split="housing")
documents = load_dataset(repo_id, "documents", split="housing")
qrels = load_dataset(repo_id, "qrels", split="housing")

Join questions.qid to qrels.qid, then join qrels.doc_id to documents.doc_id.

Sizes

SplitQuestionsDocumentsQrels
housing6,8531,837,40317,862
barexam1,195856,8351,195
kcl_essay169128,789518
legit_train24,262128,78987,423
legit_val200128,789721
legit_test300128,7891,229

The kcl-essay, legit_train, legit_val, and legit_test splits share the same 128,789-document Korean legal corpus. It is exposed under every applicable split to keep the loading interface uniform.

Schemas

  • —questions: qid, question, answer, answer_type, choices_json, eval_json, meta_json, gold_contexts_json
  • —documents: doc_id, title, text, meta_json
  • —qrels: qid, doc_id, relevance, citation, issue_id

Fields ending in _json contain JSON-encoded values. Encoding heterogeneous nested metadata this way preserves it losslessly while maintaining one stable Parquet schema across all six splits. Decode one with json.loads(value) when the value is not null. citation is populated by Housing QA; issue_id is populated by LEGIT; both are nullable elsewhere.

Benchmark notes

  • —housing: yes/no questions about U.S. state housing law as of 2021. The full retrieval corpus contains statutes from 48 states.
  • —barexam: multiple-choice MBE questions over the complete annotated passage pool. The upstream train/validation/test examples are combined here as the requested barexam split.
  • —kcl_essay: Korean bar-exam essay questions with instance-level rubrics.
  • —legit_train, legit_val, legit_test: Korean judgment prediction with legal issue-tree rubrics.

The metadata recorded with each source split is available under metadata/. The published legit_val metadata records 721 qrels, matching the file; the local precursor metadata had a stale count of 702.

legit_train contains 24,262 question rows but 24,086 distinct qid values: 176 identifiers each occur twice, with different question records. These source rows are preserved rather than silently deduplicated. Consumers that require a unique row key should combine qid with the row index; qrels continue to join on the source qid.

Sources, licenses, and attribution

This repository is a collection. The applicable license depends on the split:

Users are responsible for following the license that applies to each split. The non-commercial restriction applies to KCL and LEGIT. The Housing QA and Bar Exam QA share-alike requirements continue to apply to those materials.

Intended use and limitations

The collection is intended for research on retrieval and retrieval-augmented generation in expert legal domains. It is not legal advice. Housing questions and statutes describe law as of 2021 and may not reflect current law. Benchmark answers, annotations, and retrieved materials can be incomplete or erroneous.