infly/Infinity-Parser2-Pro

1---2license: apache-2.03language:4  - en5  - zh6  - multilingual7pipeline_tag: image-text-to-text8library_name: transformers9tags:10  - ocr11  - pdf12  - document-parsing13  - document-understanding14  - layout-analysis15  - table-recognition16  - chart-parsing17  - formula-recognition18  - chemical-formula19  - markdown20  - vision-language21  - infinity-parser22  - infinity_parser223datasets:24  - infly/Infinity-Doc2-5M25---26 27# Infinity-Parser2-Pro28 29<p align="center">30    <img src="assets/logo.png" width="400"/>31<p>32 33<p align="center">34💻 <a href="https://github.com/infly-ai/INF-MLLM">Github</a> |35📊 <a href="https://huggingface.co/datasets/infly/Infinity-Doc2-5M">Dataset</a> |36📄 <a href="https://arxiv.org/pdf/2607.07836">Paper</a> |37🚀 <a href="https://huggingface.co/spaces/infly/Infinity-Parser2-Demo">Demo</a>38</p>39 40## News41 42- [2026-05-11] Released flagship document parsing models: [Infinity-Parser2-Pro](https://huggingface.co/infly/Infinity-Parser2-Pro), [Infinity-Parser2-Flash](https://huggingface.co/infly/Infinity-Parser2-Flash), and the dataset [Infinity-Doc2-5M](https://huggingface.co/datasets/infly/Infinity-Doc2-5M). Infinity-Parser2 achieves SOTA results on olmOCR-bench and ParseBench.43 44<p align="center">45    <img src="https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/newspaper_1.png" width="1200"/>46</p>47 48## Introduction49 50We are excited to release Infinity-Parser2, our latest flagship document understanding model. We offer two distinct variants to address diverse deployment constraints: Infinity-Parser2-Pro, optimized for maximum accuracy in precision-critical tasks, achieves state-of-the-art results on olmOCR-Bench (87.6%) and ParseBench (74.3%), surpassing frontier models including DeepSeek-OCR-2, PaddleOCR-VL-1.5, and MinerU2.5. Infinity-Parser2-Flash, engineered for low-latency inference, delivers a 3.68x speedup over our previous Infinity-Parser-7B model. With significant upgrades to both our data engine and multi-task reinforcement learning approach, the model consolidates robust multi-modal parsing capabilities into a unified architecture, unlocking brand-new zero-shot capabilities across a wide range of real-world business scenarios.51  52### Key Features53 54- **Upgraded Data Engine**: We have comprehensively enhanced our synthetic data engine to support both fixed-layout and flexible-layout document formats. By curating nearly 5 million diverse document parsing samples across a wide range of layouts, combined with a dynamic adaptive sampling strategy, we ensure highly balanced and robust multi-task learning across various document types.55- **Multi-Task Reinforcement Learning**: We designed a novel verifiable reward system to support Joint Reinforcement Learning (RL), enabling seamless and simultaneous co-optimization of multiple complex tasks, including document parsing, element parsing, chart parsing, chemical formula parsing, document vqa, and general multimodal understanding.56- **Breakthrough Parsing Performance**: Infinity-Parser2-Pro substantially outperforms our previous 7B model, achieving 87.6% on olmOCR-Bench and 74.3% on ParseBench, surpassing frontier models such as DeepSeek-OCR-2, PaddleOCR-VL, and MinerU2.5.57- **Inference Acceleration**: Infinity-Parser2-Flash delivers significantly higher efficiency than Infinity-Parser-7B, with inference throughput increased by 3.68x (from 441 to 1,624 tokens/sec), reducing both deployment latency and costs.58 59## Performance60 61<p align="left">62    <img src="https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/olmocr_bench_perf.png" width="1200"/>63<p>64 65<p align="left">66    <img src="https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/parsebench_perf.png" width="1200"/>67<p>68 69<table align="center">70  <thead>71    <tr>72      <th>Task</th>73      <th>Infinity-Parser2-Pro</th>74      <th>Infinity-Parser2-Flash</th>75      <th>PaddleOCR-VL-1.5</th>76      <th>DeepSeek-OCR-2</th>77      <th>MinerU2.5</th>78      <th>Gemini-3-Pro</th>79    </tr>80  </thead>81  <tbody>82    <tr>83      <td colspan=7><b>Document Parsing</b></td>84    </tr>85    <tr>86      <td>olmOCR-bench</td>87      <td><b>87.6</b></td>88      <td>86.0</td>89      <td>78.5</td>90      <td>76.3</td>91      <td>75.2</td>92      <td>-</td>93    </tr>94    <tr>95      <td>ParseBench</td>96      <td><b>74.3</b></td>97      <td>72.2</td>98      <td>66.0</td>99      <td>41.2</td>100      <td>45.9</td>101      <td>69.1‡</td>102    </tr>103    <tr>104      <td>OmniDocBench-v1.6</td>105      <td>93.95</td>106      <td>91.98</td>107      <td><b>94.87</b></td>108      <td>90.17</td>109      <td>92.98</td>110      <td>92.85</td>111    </tr>112    <tr>113      <td colspan=7><b>Layout Analysis (mIoU)</b></td>114    </tr>115    <tr>116      <td>DocLayNet</td>117      <td>64.93*</td>118      <td>64.97*</td>119      <td><b>71.05*</b></td>120      <td>45.62*</td>121      <td>67.74*</td>122      <td>-</td>123    </tr>124    <tr>125      <td>D4LA</td>126      <td><b>52.41*</b></td>127      <td>46.05*</td>128      <td>50.21*</td>129      <td>33.03*</td>130      <td>51.62*</td>131      <td>-</td>132    </tr>133    <tr>134      <td>OmniDocBench-v1.5-Layout</td>135      <td>74.56*</td>136      <td>73.07*</td>137      <td>74.80*</td>138      <td>55.28*</td>139      <td><b>76.28*</b></td>140      <td>-</td>141    </tr>142    <tr>143      <td colspan=7><b>Element Parsing</b></td>144    </tr>145    <tr>146      <td>OmniDocBench-v1.5-TextBlock</td>147      <td><b>95.05</b></td>148      <td>94.31</td>149      <td>94.97*</td>150      <td>84.13*</td>151      <td>86.00</td>152      <td>-</td>153    </tr>154    <tr>155      <td>PubTabNet (val)</td>156      <td><b>94.76</b></td>157      <td>92.41</td>158      <td>84.60</td>159      <td>89.53*</td>160      <td>89.07</td>161      <td>91.40</td>162    </tr>163    <tr>164      <td>UniMERNet</td>165      <td><b>97.7</b></td>166      <td>96.5</td>167      <td>95.8*</td>168      <td>79.8*</td>169      <td>96.5</td>170      <td>96.4</td>171    </tr>172    <tr>173      <td colspan=7><b>Chart Parsing</b></td>174    </tr>175    <tr>176      <td>Chart2Table</td>177      <td><b>86.5</b></td>178      <td>80.49</td>179      <td>86.2*</td>180      <td>49.7*</td>181      <td>-</td>182      <td>-</td>183    </tr>184    <tr>185      <td>Chart2Json</td>186      <td><b>73.69</b></td>187      <td>67.66</td>188      <td>-</td>189      <td>-</td>190      <td>-</td>191      <td>-</td>192    </tr>193    <tr>194      <td colspan=7><b>Chemical Formula Parsing</b></td>195    </tr>196    <tr>197      <td>CoSyn_Chemical</td>198      <td><b>73.19</b></td>199      <td>63.34</td>200      <td>-</td>201      <td>47.02*</td>202      <td>-</td>203      <td>-</td>204    </tr>205    <tr>206      <td colspan=7><b>Document VQA</b></td>207    </tr>208    <tr>209      <td>DocVQA (val)</td>210      <td><b>96.43</b></td>211      <td>93.16</td>212      <td>-</td>213      <td>43.42*</td>214      <td>-</td>215      <td>93.68*</td>216    </tr>217    <tr>218      <td>InfoVQA (val)</td>219      <td><b>86.26</b></td>220      <td>75.94</td>221      <td>-</td>222      <td>22.07*</td>223      <td>-</td>224      <td>85.24*</td>225    </tr>226    <tr>227      <td colspan=7><b>General Multimodal Understanding</b></td>228    </tr>229    <tr>230      <td>AI2D</td>231      <td>88.89</td>232      <td>79.53</td>233      <td>-</td>234      <td>37.66*</td>235      <td>-</td>236      <td><b>91.87*</b></td>237    </tr>238    <tr>239      <td>MathVista (testmini)</td>240      <td>71.4</td>241      <td>59.5</td>242      <td>-</td>243      <td>-</td>244      <td>-</td>245      <td><b>81.8*</b></td>246    </tr>247    <tr>248      <td>MMBench-EN (dev)</td>249      <td>87.54</td>250      <td>77.92</td>251      <td>-</td>252      <td>-</td>253      <td>-</td>254      <td><b>90.29*</b></td>255    </tr>256    <tr>257      <td>MMBench-CN (dev)</td>258      <td>86.43</td>259      <td>75.77</td>260      <td>-</td>261      <td>-</td>262      <td>-</td>263      <td><b>90.98*</b></td>264    </tr>265    <tr>266      <td>MMMU (val)</td>267      <td><b>61.89</b></td>268      <td>45.89</td>269      <td>-</td>270      <td>-</td>271      <td>-</td>272      <td>56.00*</td>273    </tr>274    <tr>275      <td>MMStar</td>276      <td>69.66</td>277      <td>57.13</td>278      <td>-</td>279      <td>-</td>280      <td>-</td>281      <td><b>83.78*</b></td>282    </tr>283    <tr>284      <td>OCRBench</td>285      <td>86.20</td>286      <td>81.60</td>287      <td>-</td>288      <td>47.20*</td>289      <td>-</td>290      <td><b>89.30*</b></td>291    </tr>292  </tbody>293</table>294 295Note: '*' denotes results evaluated using our internal evaluation tools. '‡' denotes results from the Gemini-3.1-Pro.296 297## Quick Start298 299### 1. Minimal "Hello World" (Native Transformers)300 301If you are looking for a minimal script to parse a single image to Markdown using the native `transformers` library, here is a simple snippet:302 303```python304from PIL import Image305import torch306from transformers import AutoModelForImageTextToText, AutoProcessor307from qwen_vl_utils import process_vision_info308 309# Load the model and processor310model = AutoModelForImageTextToText.from_pretrained(311    "infly/Infinity-Parser2-Pro",312    torch_dtype="float16",313    device_map="auto",314)315processor = AutoProcessor.from_pretrained("infly/Infinity-Parser2-Pro")316 317# Build the messages for the model318pil_image = Image.open("demo_data/demo.png").convert("RGB")319min_pixels = 2048  # 32 * 64320max_pixels = 16777216  # 4096 * 4096321prompt = """322- Extract layout information from the provided PDF image.323- For each layout element, output its bbox, category, and the text content within the bbox.324- Bbox format: [x1, y1, x2, y2].325- Allowed layout categories: ['header', 'title', 'text', 'figure', 'table', 'formula', 'figure_caption', 'table_caption', 'formula_caption', 'figure_footnote', 'table_footnote', 'page_footnote', 'footer'].326- Text extraction and formatting:327  1) For 'figure', the text field must be an empty string.328  2) For 'formula', format text as LaTeX.329  3) For 'table', format text as HTML.330  4) For all other categories (e.g., text, title), format text as Markdown.331- The output text must be exactly the original text from the image, with no translation or rewriting.332- Sort all layout elements in human reading order.333- Final output must be a single JSON object.334"""335 336messages = [337    {338        "role": "user",339        "content": [340            {341                "type": "image",342                "image": pil_image,343                "min_pixels": min_pixels,344                "max_pixels": max_pixels,345            },346            {"type": "text", "text": prompt},347        ],348    }349]350 351chat_template_kwargs = {"enable_thinking": False}352 353text = processor.apply_chat_template(354    messages, tokenize=False, add_generation_prompt=True, **chat_template_kwargs355)356image_inputs, _ = process_vision_info(messages, image_patch_size=16)357 358inputs = processor(359    text=text,360    images=image_inputs,361    do_resize=False,362    padding=True,363    return_tensors="pt",364)365 366# Move all tensors to the same device as the model367inputs = {368    k: v.to(model.device) if isinstance(v, torch.Tensor) else v369    for k, v in inputs.items()370}371 372# Generate the response373generated_ids = model.generate(374    **inputs,375    max_new_tokens=32768,376    temperature=0.0,377    top_p=1.0,378)379 380# Strip input tokens, keeping only the newly generated response381generated_ids_trimmed = [382    out_ids[len(in_ids) :]383    for in_ids, out_ids in zip(inputs["input_ids"], generated_ids)384]385output_text = processor.batch_decode(386    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False387)388print(output_text)389```390 391### 2. Advanced Pipeline (infinity_parser2)392 393For bulk processing, advanced features, or an end-to-end PDF parsing pipeline, we recommend using our infinity_parser2 wrapper.394 395#### Pre-requisites396 397```bash398# Create a Conda environment (Optional)399conda create -n infinity_parser2 python=3.12400conda activate infinity_parser2401 402# Install PyTorch (CUDA). Find the proper version at https://pytorch.org/get-started/previous-versions based on your CUDA version.403pip install torch==2.10.0 torchvision==0.25.0 torchaudio==2.10.0 --index-url https://download.pytorch.org/whl/cu128404 405# Install FlashAttention (FlashAttention-2 is recommended by default)406# Standard install (compiles from source, ~10-30 min):407pip install flash-attn==2.8.3 --no-build-isolation408# Faster install: download wheel from https://github.com/Dao-AILab/flash-attention/releases. Then run: pip install /path/to/<wheel_filename>.whl409# For Hopper GPUs (e.g. H100, H800), we recommend FlashAttention-3 instead. See: https://github.com/Dao-AILab/flash-attention410# NOTE: The code will prioritize detecting FlashAttention-3. If not found, it falls back to FlashAttention-2.411 412# Install vLLM413# NOTE: you may need to run the command below to resolve triton and numpy conflicts before installing vllm.414# pip uninstall -y pytorch-triton opencv-python opencv-python-headless numpy && rm -rf "$(python -c 'import site; print(site.getsitepackages()[0])')/cv2"415pip install vllm==0.17.1416```417 418#### Install infinity_parser2419 420Install from PyPI421 422```bash423pip install infinity_parser2424```425 426Install from source code427 428```bash429git clone https://github.com/infly-ai/INF-MLLM.git430cd INF-MLLM/Infinity-Parser2431pip install -e .432```433 434#### Usage435 436##### Command Line437 438The `parser` command is the fastest way to get started.439 440```bash441# NOTE: The Infinity-Parser2 model will be automatically downloaded on the first run.442 443# Parse a PDF (outputs Markdown by default)444parser demo_data/demo.pdf445 446# Parse an image447parser demo_data/demo.png448 449# Batch parse multiple files450parser demo_data/demo.pdf demo_data/demo.png -o ./output451 452# Parse an entire directory453parser demo_data -o ./output454 455# Output raw JSON with layout bboxes456parser demo_data/demo.pdf --output-format json457 458# Convert to Markdown directly459parser demo_data/demo.png --task doc2md460```461 462```bash463# View all options464parser --help465```466 467##### Python API468 469```python470# NOTE: The Infinity-Parser2 model will be automatically downloaded on the first run.471 472from infinity_parser2 import InfinityParser2473 474parser = InfinityParser2()475 476# Parse a single file (returns Markdown)477result = parser.parse("demo_data/demo.pdf")478print(result)479 480# Parse multiple files (returns list)481results = parser.parse(["demo_data/demo.pdf", "demo_data/demo.png"])482 483# Parse a directory (returns dict)484results = parser.parse("demo_data")485```486 487**Output formats:**488 489| task_type   | Description                                          | Default Output |490|-------------|------------------------------------------------------|----------------|491| `doc2json`  | Extract layout elements with bboxes (default)        | Markdown       |492| `doc2md`    | Directly convert to Markdown                         | Markdown       |493| `custom`    | Use your own prompt                                 | Raw model output |494 495```python496# doc2json: get raw JSON with bbox coordinates497result = parser.parse("demo_data/demo.pdf", output_format="json")498 499# doc2md: direct Markdown conversion500result = parser.parse("demo_data/demo.pdf", task_type="doc2md")501 502# Custom prompt503result = parser.parse("demo_data/demo.pdf", task_type="custom",504                      custom_prompt="Please transform the document's contents into Markdown format.")505 506# Batch processing with custom batch size507result = parser.parse("demo_data", batch_size=8)508 509# Save results to directory510parser.parse("demo_data/demo.pdf", output_dir="./output")511```512 513**Backends:**514 515Infinity-Parser2 supports three inference backends. By default it uses the **vLLM Engine** (offline batch inference).516 517```python518# vLLM Engine (default) — offline batch inference519parser = InfinityParser2(520    model_name="infly/Infinity-Parser2-Pro",521    backend="vllm-engine",        # default522    tensor_parallel_size=2,523)524 525# Transformers — local single-GPU inference526parser = InfinityParser2(527    model_name="infly/Infinity-Parser2-Pro",528    backend="transformers",529    device="cuda",530    torch_dtype="bfloat16",       # "float16" or "bfloat16"531)532 533# vLLM Server — online HTTP API (start server first)534parser = InfinityParser2(535    model_name="infly/Infinity-Parser2-Pro",536    backend="vllm-server",537    api_url="http://localhost:8000/v1/chat/completions",538    api_key="EMPTY",539)540```541 542To start a vLLM server:543 544```bash545vllm serve infly/Infinity-Parser2-Pro \546    --trust-remote-code \547    --reasoning-parser qwen3 \548    --host 0.0.0.0 \549    --port 8000 \550    --tensor-parallel-size 2 \551    --gpu-memory-utilization 0.85 \552    --max-model-len 65536 \553    --mm-encoder-tp-mode data \554    --mm-processor-cache-type shm \555    --enable-prefix-caching556```557 558For more details, please refer to the [official guide](https://github.com/infly-ai/INF-MLLM/blob/main/Infinity-Parser2).559 560### Visual Parsing Examples561 562| Visualization | Note |563| --- | --- | 564| [A-Stock](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/a_stock.png) | Easy to miscount colspan in tables |565| [Multi-Column Layout](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/muti_column.png) | Complex layout analysis and reading order recovery. |566| [Historical Newspaper](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/newspaper_2.png) | High probability of bounding box omission caused by ultra-dense text distribution, narrow column margins, and microscopic fonts. |567| [US-Stock](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/us_stock.png) | Precise row alignment across wide frameless spaces and capturing the hierarchical semantics of indented headers. |568| [Academic Paper (arXiv)](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/arxiv.png) | Accurate structural preservation of complex multi-line mathematical formulas, dense inline notations, and deeply nested subscripts/superscripts. |569| [Magazine Page](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/magazine.png) | Complex reading order recovery in an asymmetric multi-column layout. |570| [Scanned Mathematics](https://raw.githubusercontent.com/infly-ai/INF-MLLM/main/Infinity-Parser2/assets/old_scan_math.png) | Degraded and blurred print |571 572## Limitations573 574Infinity-Parser2 has several known limitations to consider. It primarily supports English and Chinese documents, and performance degrades when processing multilingual content. Accuracy may also be reduced when parsing charts with complex layouts, as well as documents containing multi-oriented elements such as table rotated at varying angles. Additionally, the model does not capture fine-grained text formatting (e.g., bold, italic, strikethrough) and exhibits suboptimal multimodal instruction-following capability, meaning it may not always reliably follow complex multi-step visual instructions.575 576## Acknowledgments577 578We would like to thank [Qwen3.5](https://github.com/QwenLM/Qwen3.5), [ms-swift](https://github.com/modelscope/ms-swift), [VeRL](https://github.com/verl-project/verl), [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval), [olmocr](https://huggingface.co/datasets/allenai/olmOCR-bench), [PaddleOCR-VL](https://github.com/PaddlePaddle/PaddleOCR), [MinerU](https://github.com/opendatalab/MinerU), [dots.ocr](https://github.com/rednote-hilab/dots.ocr), [Chandra-OCR-2](https://github.com/datalab-to/chandra) for providing dataset, code and models.579 580## Citation581 582```583@misc{huang2026infinityparser2technicalreport,584      title={Infinity-Parser2 Technical Report},585      author={Zuming Huang and Jun Huang and Kexuan Ren and Baode Wang and Weizhen Li and Jianming Feng and Yu Wang and Yichen Yao and Shijun Lin and Yige Tang and Cheng Peng and Weidi Xu and Wei Chu and Yinghui Xu and Yuan Qi},586      year={2026},587      eprint={2607.07836},588      archivePrefix={arXiv},589      primaryClass={cs.AI},590      url={https://arxiv.org/abs/2607.07836},591}592```593 594# License595 596This model is licensed under apache-2.0.