Datasetpublic

CZLC/history_retrieval

Introduction The HistoryIR dataset was annotated on top of the historical part of BUT-LCC corpus. We urged annotators to search for historical events (from their own mind, or using our inspirator, more details in the upcoming paper), using the semantic search tool we developed (translation service + English contriever model setup). Then the annotators annotated top retrieved passages as relevant or irrelevant. We've done additional filtering step that included manual… See the full description on the dataset page: https://huggingface.co/datasets/CZLC/history_retrieval.

sourceHugging Facecc-by-4.0updated 2y agoView on Hugging Face

0likes85downloads

Dataset Card

Introduction

The HistoryIR dataset was annotated on top of the historical part of BUT-LCC corpus. We urged annotators to search for historical events (from their own mind, or using our inspirator, more details in the upcoming paper), using the semantic search tool we developed (translation service + English contriever model setup). Then the annotators annotated top retrieved passages as relevant or irrelevant.

We've done additional filtering step that included manual verification of several annotators of every annotator. We've only included annotations from the given time period (specific to the annotator), which passed the manual verification test.

Creating Multi-Choice format

To create multi-choice format, we sample quadruplets of relevant/irrelevant documents for each query such that

—3 documents were irrelevant
—1 document was relevant

—3 documents were relevant
—1 document was irrelevant.

The task then is the identify the incosisistent document (e.g. identify single relevant from 4, or identify single irrelevant from 4). The multi-choice format conversion script is available in the repository as convert_histir_filtered.py.

Licensing

The historical documents are not owned by CZLC affiliated members and belong to the original authors. The annotations are released under CC-BY 4.0 license and extra code is released under Apache-2.0 licensing.

Citation

If you use this dataset, please cite the following bibtex

@misc{fajcik2024czech,
  author       = {Martin Fajčík and Martin Dočekal and Jakub Štetina and Michal Hradiš},
  title        = {HistIrCzech: A Czech Corpus for Historical Document Retrieval},
  year         = {2024},
  howpublished = {\url{https://huggingface.co/datasets/CZLC/history_retrieval}},
  note         = {Dataset published on Hugging Face}
}

We still seek to extend the size and variability of our IR dataset before it's final publication.

Additional statistics

Dataset size: 1196
Average positive (1 pos, 3 neg) word length: 100.05287713841369
Average negative (1 pos, 3 neg) word length: 84.81804043545878
Average positive (3 pos, 1 neg) word length: 95.6931886678722
Average negative (3 pos, 1 neg) word length: 85.69620253164557
3pos_1neg examples: 553
1pos_3neg examples: 643