songjhPKU/PM4Bench
PM4Bench Strictly parallel multilingual evaluation for Large Vision-Language Models Overview The paper Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning introduces PM4Bench to separate language effects from dataset variation. Its content is strictly parallel across ten languages, and its vision setting renders textual inputs directly into images. Comparing that setting with interleaved input identifies OCR… See the full description on the dataset page: https://huggingface.co/datasets/songjhPKU/PM4Bench.
<div align="center">
PM4Bench
Strictly parallel multilingual evaluation for Large Vision-Language Models
    
</div>
Overview
The paper *Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning* introduces PM4Bench to separate language effects from dataset variation. Its content is strictly parallel across ten languages, and its vision setting renders textual inputs directly into images. Comparing that setting with interleaved input identifies OCR as a key source of cross-lingual performance gaps and motivates the QGO training method linked above.
This repository contains the PM4Bench evaluation data spanning MDUR, MIQA, MSOCR, and MGUI. Version 2 stores each image once and references it from portable JSONL manifests; the legacy TSV/base64 format is retired.
Components
The repository-level Apache-2.0 declaration applies to PM4Bench-owned metadata, translations, renderings, templates, and annotations. MMMU-Pro content in MDUR remains under Apache-2.0. MMDU-derived content in MIQA remains under CC BY-NC 4.0. Upstream components retain their attribution and applicable terms; see UPSTREAM_LICENSES.md and the upstream dataset cards for their complete notices.
Formats and integrity
The dataset contains 40 UTF-8 JSONL manifests and portable relative asset paths. It contains no TSV or embedded-base64 table. metadata/release_inventory.json records the language/task counts, and SHA256SUMS authenticates released files.
Loading
Use snapshot_download to retain relative asset paths, then read the selected JSONL files. The pm4bench toolkit validates paths and provides metrics.
from huggingface_hub import snapshot_download
root = snapshot_download(repo_id="songjhPKU/PM4Bench", repo_type="dataset")The PM4Bench codebase provides schemas, validation, evaluation, construction tools, and the QGO recipe.
MGUI construction note
The released MGUI images and GT are canonical. They were produced with Jinja2 3.1.6, Selenium 4.40.0, and Chrome for Testing 119.0.6045.105. A historical Jinja autoescaping interaction caused CSS font-family declarations to fall back to platform fonts, so fresh rendering on another OS can change bounding boxes. The toolkit's structure comparison verifies all 1,000 pages, DOM targets, visible text, questions, and metadata without replacing canonical geometry; strict mode additionally audits bbox drift.
Cite Us
<pre><code class="language-bibtex"> @misc{gao2026benchmarkingboostingmultilingualcapabilities, title={Benchmarking and Boosting Multilingual Capabilities of LVLMs via OCR-Centric Reinforcement Learning}, author={Junyuan Gao and Jiahe Song and Jiang Wu and Runchuan Zhu and Guanlin Shen and Shasha Wang and Xingjian Wei and Haote Yang and Weijia Li and Bin Wang and Lijun Wu and Conghui He}, year={2026}, eprint={2503.18484}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2503.18484v3}, } </code></pre>