Datasetpublic

omarelba/letzcross-wiki-parallel

LëtzCross Wiki Parallel Dataset Description omarelba/letzcross-wiki-parallel is a multilingual visual-document-retrieval dataset. Each row pairs a rendered PDF page with parallel questions in English, French, German, and, where available, Luxembourgish. The dataset also includes answer and provenance fields produced during dataset construction and validation. The dataset is used for language-specific late-interaction page-image retrieval training. A training… See the full description on the dataset page: https://huggingface.co/datasets/omarelba/letzcross-wiki-parallel.

sourceHugging Faceupdated 27d agoView on Hugging Face

0likes134downloads

Dataset Card

LëtzCross Wiki Parallel

Dataset Description

omarelba/letzcross-wiki-parallel is a multilingual visual-document-retrieval dataset. Each row pairs a rendered PDF page with parallel questions in English, French, German, and, where available, Luxembourgish. The dataset also includes answer and provenance fields produced during dataset construction and validation.

The dataset is used for language-specific late-interaction page-image retrieval training. A training sample consists of one language-specific question as the query and image as the positive document target.

Dataset Structure

Data Instances

python

{
    "id": 1143,
    "query_en": "On what date did Joseph Lorenzo Philippe die in Luxembourg after a long illness?",
    "query_fr": "À quelle date Joseph Lorenzo Philippe est-il décédé au Luxembourg après une longue maladie?",
    "query_de": "An welchem Datum starb Joseph Lorenzo Philippe in Luxemburg nach langer Krankheit?",
    "query_lb": None,
    "image": <PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=421x553>,
    "filename": "Joseph Lorenzo Philippe.pdf",
    "page": 1,
    "gold_answer": "21. Oktober 1956",
    "generated_answer": "21. Oktober 1956",
    "source": "text",
    "exact_match": True,
    "semantic_match": True,
}

Data Fields

Field	Type	Description
`id`	`int64`	Identifier for the underlying question/document example.
`query_en`	`string`	English version of the question.
`query_fr`	`string`	French version of the same question.
`query_de`	`string`	German version of the same question.
`query_lb`	`string`	Luxembourgish version of the same question. This field may be null.
`image`	`Image`	Rendered image of the relevant PDF page; used as the positive retrieval target.
`filename`	`string`	Filename of the source PDF or article document.
`page`	`int64`	Page number of the rendered source page.
`gold_answer`	`string`	Reference answer supported by the page.
`generated_answer`	`string`	Answer generated or extracted during dataset construction and validation.
`source`	`string`	Evidence type supporting the answer, for example `text` or `table`.
`exact_match`	`bool`	Whether `generated_answer` exactly matches `gold_answer`.
`semantic_match`	`bool`	Whether the generated and reference answers were judged semantically equivalent.

Data Splits

Split	Rows
`train`	22,028
`validation`	450
Total	22,478

Usage

python

from datasets import load_dataset

dataset = load_dataset("omarelba/letzcross-wiki-parallel")
train_dataset = dataset["train"]

Citation

If you use this dataset, please cite:

LëtzCross: A Cross-Lingual Page-Level Benchmark for Multimodal Retrieval over Luxembourgish Documents

https://arxiv.org/abs/2608.21714