datasets
Training and evaluation data, with the modality, task and licence stated up front. Listed live from the Hugging Face Hub.
VisRAG-Ret-Train-Synthetic-data
Dataset Description
This dataset is the synthetic part of the training set of VisRAG it includes 239,358 Query-Document (Q-D) Pairs from a synthetic dataset made up
of pages from web-crawled PDF documents and augmented with VLM-generated (GPT-4o) pseudo-queries.
Our training data is organized with a batch size of 128, ensuring that all data within the same batch comes from the same dataset.
Name
Source
Description
# Pages
Textbooks
https://openstax.org/
College-level… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data.RLAIF-V-Dataset
Dataset Card for RLAIF-V-Dataset
This dataset was introduced in RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthiness.
GitHub
This dataset was also used in MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe
News:
[2025.09.18] 🎉 Our data is used in the powerful MiniCPM-V 4.5 model, which represents a state-of-the-art end-side MLLM achieving GPT-4o level performance!
[2025.03.01] 🎉 RLAIF-V is accepted by CVPR… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/RLAIF-V-Dataset.VisRAG-Ret-Test-ArxivQA
Dataset Description
This is a VQA dataset based on figures extracted from arXiv publications taken from ArXiVQA dataset from Multimodal ArXiV.
Load the dataset
from datasets import load_dataset
import csv
def load_beir_qrels(qrels_file):
qrels = {}
with open(qrels_file) as f:
tsvreader = csv.DictReader(f, delimiter="\t")
for row in tsvreader:
qid = row["query-id"]
pid = row["corpus-id"]
rel = int(row["score"])… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-ArxivQA.VisRAG-Ret-Test-SlideVQA
Dataset Description
This is a VQA dataset based on Slide Decks from SlideVQA dataset from SlideVQA.
Load the dataset
from datasets import load_dataset
import csv
def load_beir_qrels(qrels_file):
qrels = {}
with open(qrels_file) as f:
tsvreader = csv.DictReader(f, delimiter="\t")
for row in tsvreader:
qid = row["query-id"]
pid = row["corpus-id"]
rel = int(row["score"])
if qid in qrels:… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-SlideVQA.VisRAG-Ret-Test-PlotQA
Dataset Description
This is a VQA dataset based on Scientific Plots from PlotQA dataset from PlotQA.
Load the dataset
from datasets import load_dataset
import csv
def load_beir_qrels(qrels_file):
qrels = {}
with open(qrels_file) as f:
tsvreader = csv.DictReader(f, delimiter="\t")
for row in tsvreader:
qid = row["query-id"]
pid = row["corpus-id"]
rel = int(row["score"])
if qid in qrels:… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-PlotQA.VisRAG-Ret-Test-ChartQA
Dataset Description
This is a VQA dataset based on Charts from ChartQA dataset from ChartQA.
Load the dataset
from datasets import load_dataset
import csv
def load_beir_qrels(qrels_file):
qrels = {}
with open(qrels_file) as f:
tsvreader = csv.DictReader(f, delimiter="\t")
for row in tsvreader:
qid = row["query-id"]
pid = row["corpus-id"]
rel = int(row["score"])
if qid in qrels:… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-ChartQA.VisRAG-Ret-Test-MP-DocVQA
Dataset Description
This is a VQA dataset based on Industrial Documents from MP-DocVQA dataset from MP-DocVQA.
Load the dataset
from datasets import load_dataset
import csv
def load_beir_qrels(qrels_file):
qrels = {}
with open(qrels_file) as f:
tsvreader = csv.DictReader(f, delimiter="\t")
for row in tsvreader:
qid = row["query-id"]
pid = row["corpus-id"]
rel = int(row["score"])
if qid in qrels:… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-MP-DocVQA.VisRAG-Ret-Test-InfoVQA
Dataset Description
This is a VQA dataset based on Infographics from InfoVQA dataset from InfoVQA.
Load the dataset
from datasets import load_dataset
import csv
def load_beir_qrels(qrels_file):
qrels = {}
with open(qrels_file) as f:
tsvreader = csv.DictReader(f, delimiter="\t")
for row in tsvreader:
qid = row["query-id"]
pid = row["corpus-id"]
rel = int(row["score"])
if qid in qrels:… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Test-InfoVQA.VisRAG-Ret-Train-In-domain-data
Dataset Description
This dataset is the In-domain part of the training set of VisRAG it includes 122,752 Query-Document (Q-D) Pairs from openly available academic datasets.
Our training data is organized with a batch size of 128, ensuring that all data within the same batch comes from the same dataset.
Dataset
# Q-D Pairs
ArXivQA
25,856
ChartQA
4,224
MP-DocVQA
10,624
InfoVQA
17,664
PlotQA
56,192
SlideVQA
8,192
Load the dataset
from datasets import… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data.RLHF-V-Dataset
Dataset Card for RLHF-V-Dataset
Project Page | Paper | GitHub
Updates
[2024.05.28] 📃 Our RLAIF-V paper is accesible at arxiv now!
[2024.05.20] 🎉 We release a new feedback dataset, RLAIF-V-Dataset, which is a large-scale diverse-task multimodal feedback dataset constructed using open-source models. You can download the corresponding dataset and models (7B, 12B) now!
[2024.04.11] 🔥 Our data is used in MiniCPM-V 2.0, an end-side multimodal large language model that… See the full description on the dataset page: https://huggingface.co/datasets/openbmb/RLHF-V-Dataset.DemoCaseopenbmb_VisRAG-Ret-Train-In-domain-datastructured-generation-information-extraction-vlms-openbmb-RLAIF-V-Datasetstructured-generation-information-extraction-vlms-openbmb-RLAIF-V-Datasetstructured-generation-information-extraction-vlms-openbmb-RLAIF-V-Datasetstructured-generation-information-extraction-vlms-openbmb-RLAIF-V-Dataset