UlrickBL/vidore_benchmark_2_biomedical_lectures_v2_reranker_adapted

Dataset Card for Vidore Reranker Benchmark : vidorebenchmark2biomedicallecturesv2reranker_adapted

Dataset Summary

This dataset provides a reranking benchmark based on the VIDORE V2 benchmark, designed to evaluate reranker models in a multimodal retrieval context. The dataset includes a corpus of image data, a set of natural language queries, and the top 25 retrievals (images) returned by a mid-performance multimodal retriever. This setup simulates a realistic retrieval environment where the reranker must learn to surface relevant items that may not already be ranked highly.

Complete benchmark dataset list

The benchmark is composed of those datasets :

• —UlrickBL/vidorebenchmarkeconomicsreportsv2rerankeradapted (linked to vidore/economicsreportsv2 corpus)
• —UlrickBL/vidorebenchmarkdocvqarerankeradapted (linked to vidore/docvqatestsubsampled corpus)
• —UlrickBL/vidorebenchmark2biomedicallecturesv2rerankeradapted (linked to vidore/biomedicallectures_v2 corpus)
• —UlrickBL/vidorebenchmark2esgreportshumanlabeledv2rerankeradapted (linked to vidore/esgreportshumanlabeled_v2 corpus)
• —UlrickBL/vidorebenchmark2esgreportsv2rerankeradapted (linked to vidore/esgreports_v2 corpus)

Dataset Motivation

The purpose of this benchmark is to:

• —Evaluate rerankers independently of retriever performance by fixing the retriever outputs.

• —Focus on the effectiveness of rerankers in identifying relevant samples from mid-quality retrieval sets.

• —Provide detailed statistics on the retrieval and relevance structure to better understand model behavior.

By using a retriever with known mid-level performance on the VIDORE v1 leaderboard, this benchmark offers a challenging but meaningful setting to test reranking capabilities.

The retriever used is : Alibaba-NLP/gme-Qwen2-VL-2B-Instruct (top 23 - 87.8 accuracy)

Dataset Structure

Each sample in the dataset is associated with:

• —query : text query

• —query_id : corresponding query id of the original dataset

• —top25image_filenames : ordered list of retrieved image by the retriever

• —top25scores : corresponding list with the scores of the top 25 images

• —relevantimagefilenames : list of the true labels / relevant images of the original dataset

How the Dataset Was Constructed

Retriever Used: A multimodal retriever Alibaba-NLP/gme-Qwen2-VL-2B-Instruct

The retriever was used to embed the full corpus of images of the associated dataset (vidore/biomedicallecturesv2).

For each query, the retriever computed similarity and returned the top 25 most similar corpus images.

These 25 candidates were labeled using the ground-truth relevance annotations from VIDORE v2.

Only retrieved items are considered during evaluation — non-retrieved relevant samples are ignored to focus on reranking.

Dataset Statistics

Here are some key dataset statistics:

Use this dataset

To use this dataset, you can create pairs of queries and images by linking a query and an image from the corpus of the top 25 list and score it with your model to rerank the top 25 list.