ufca-llms/Ulysses-RFCorpus

Ulysses-RFCorpus is a Brazilian Portuguese corpus for legislative LIR with explicit relevance feedback collected in a real production scenario at the Brazilian Chamber of Deputies. The corpus was designed to capture user feedback from the institution's own retrieval workflow, rather than synthetic labels.

The Brazilian Chamber of Deputies includes a specialized department, Legislative Consulting, which supports parliamentarians during the law-making process. In this workflow, parliamentarians can request this department to draft new bills. The queries in Ulysses-RFCorpus correspond to these requests, anonymized, and reflect real information needs. The document collection is composed of legislative proposals (bills). The relevance judgments were provided by 54 legislative consultants using a three-level scale over a pool of retrieved documents, and they could also manually indicate additional relevant bills.

According to the authors, the resource is especially relevant because publicly available legal corpora with expert or user-provided relevance feedback are still scarce, particularly in the legislative domain. The collection includes 692 queries with relevance judgments, reflecting diverse legislative information needs and real user feedback patterns captured during operational retrieval sessions. In this sense, Ulysses-RFCorpus complements predominantly judicial benchmarks by providing supervision aligned with legislative drafting and policy-support tasks.

Within JUÁ, this subset brings relevance judgments grounded in the real operational needs of legislative consultants. This improves ecological validity and helps evaluate whether models generalize beyond lexical matching to the kind of judgments observed in practical legal-government use. The adapted dataset is publicly available on Hugging Face.