aakash-projects/biomedical_lectures_eng_v2

Vidore Benchmark 2 - MIT Dataset

This dataset is part of the "Vidore Benchmark 2" collection, designed for evaluating visual retrieval applications. It focuses on the theme of MIT courses in anatomy (precisely tissue interactions).

Dataset Summary

Each query is in english.

This dataset provides a focused benchmark for visual retrieval tasks related to MIT biology courses. It includes a curated set of documents, queries, relevance judgments (qrels), and page images.

• —Number of Documents: 27
• —Number of Queries: 160
• —Number of Pages: 1016
• —Number of Relevance Judgments (qrels): 515
• —Average Number of Pages per Query: 3.2

Dataset Structure (Hugging Face Datasets)

The dataset is structured into the following columns:

• —`docs`: Contains document metadata, likely including a "doc-id" field to uniquely identify each document.
• —`corpus`: Contains page-level information:
• —"image": The image of the page (a PIL Image object).
• —"doc-id": The ID of the document this page belongs to.
• —"corpus-id": A unique identifier for this specific page within the corpus.
• —`queries`: Contains query information:
• —"query-id": A unique identifier for the query.
• —"query": The text of the query.
• —`qrels`: Contains relevance judgments:
• —"corpus-id": The ID of the relevant page.
• —"query-id": The ID of the query.
• —"answer": Answer relevant to the query AND the page.
• —"score": The relevance score.

Usage

This dataset is designed for evaluating the performance of visual retrieval systems, particularly those focused on document image understanding.

Example Evaluation with ColPali (CLI):

Here's a code snippet demonstrating how to evaluate the ColPali model on this dataset using the vidore-benchmark command-line tool.

1. 1.Install the `vidore-benchmark` package:

    pip install vidore-benchmark datasets

1. 1.Run the evaluation:

    vidore-benchmark evaluate-retriever \
        --model-class colpali \
        --model-name vidore/colpali-v1.3 \
        --dataset-name vidore/synthetic_mit_biomedical_tissue_interactions_unfiltered \
        --dataset-format beir \
        --split test

For more details on using vidore-benchmark, refer to the official documentation: https://github.com/illuin-tech/vidore-benchmark

Citation

If you use this dataset in your research or work, please cite:

@misc{faysse2024colpaliefficientdocumentretrieval,
  title={ColPali: Efficient Document Retrieval with Vision Language Models}, 
  author={Manuel Faysse and Hugues Sibille and Tony Wu and Bilel Omrani and Gautier Viaud and Céline Hudelot and Pierre Colombo},
  year={2024},
  eprint={2407.01449},
  archivePrefix={arXiv},
  primaryClass={cs.IR},
  url={https://arxiv.org/abs/2407.01449}, 
}

@misc{macé2025vidorebenchmarkv2raising,
    title={ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval}, 
    author={Quentin Macé and António Loison and Manuel Faysse},
    year={2025},
    eprint={2505.17166},
    archivePrefix={arXiv},
    primaryClass={cs.IR},
    url={https://arxiv.org/abs/2505.17166}, 
}

Acknowledgments

This work is partially supported by ILLUIN Technology, and by a grant from ANRT France.