algoryn/dots.ocr

1---2license: mit3library_name: dots_ocr4pipeline_tag: image-text-to-text5tags:6- image-to-text7- ocr8- document-parse9- layout10- table11- formula12language:13- en14- zh15- multilingual16---17 18<div align="center">19 20<p align="center">21    <img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/logo.png" width="300"/>22<p>23 24<h1 align="center">25dots.ocr: Multilingual Document Layout Parsing in a Single Vision-Language Model26</h1>27 28[![Blog](https://img.shields.io/badge/Blog-View_on_GitHub-333.svg?logo=github)](https://github.com/rednote-hilab/dots.ocr/blob/master/assets/blog.md)29[![HuggingFace](https://img.shields.io/badge/HuggingFace%20Weights-black.svg?logo=HuggingFace)](https://huggingface.co/rednote-hilab/dots.ocr)30 31 32<div align="center">33  <a href="https://dotsocr.xiaohongshu.com" target="_blank" rel="noopener noreferrer"><strong>🖥️ Live Demo</strong></a> | 34  <a href="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/wechat.jpg" target="_blank" rel="noopener noreferrer"><strong>💬 WeChat</strong></a> | 35  <a href="https://www.xiaohongshu.com/user/profile/683ffe42000000001d021a4c" target="_blank" rel="noopener noreferrer"><strong>📕 rednote</strong></a>36</div>37 38</div>39 40 41 42## Introduction43 44**dots.ocr** is a powerful, multilingual document parser that unifies layout detection and content recognition within a single vision-language model while maintaining good reading order. Despite its compact 1.7B-parameter LLM foundation, it achieves state-of-the-art(SOTA) performance.45 461. **Powerful Performance:** **dots.ocr** achieves SOTA performance for text, tables, and reading order on [OmniDocBench](https://github.com/opendatalab/OmniDocBench), while delivering formula recognition results comparable to much larger models like Doubao-1.5 and gemini2.5-pro.472. **Multilingual Support:** **dots.ocr** demonstrates robust parsing capabilities for low-resource languages, achieving decisive advantages across both layout detection and content recognition on our in-house multilingual documents benchmark.483. **Unified and Simple Architecture:** By leveraging a single vision-language model, **dots.ocr** offers a significantly more streamlined architecture than conventional methods that rely on complex, multi-model pipelines. Switching between tasks is accomplished simply by altering the input prompt, proving that a VLM can achieve competitive detection results compared to traditional detection models like DocLayout-YOLO.494.  **Efficient and Fast Performance:** Built upon a compact 1.7B LLM, **dots.ocr** provides faster inference speeds than many other high-performing models based on larger foundations.50 51 52### Performance Comparison: dots.ocr vs. Competing Models53<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/chart.png" border="0" />54 55> **Notes:** 56> - The EN, ZH metrics are the end2end evaluation results of [OmniDocBench](https://github.com/opendatalab/OmniDocBench), and Multilingual metric is the end2end evaluation results of dots.ocr-bench.57 58 59## News 60* ```2025.07.30 ``` 🚀 We release [dots.ocr](https://github.com/rednote-hilab/dots.ocr), — a multilingual documents parsing model based on 1.7b llm, with SOTA performance.61 62 63 64## Benchmark Results65 66### 1. OmniDocBench67 68#### The end-to-end evaluation results of different tasks.69 70<table>71<thead>72<tr>73<th rowspan="2"><strong>Model<br>Type</strong></th>74<th rowspan="2"><strong>Methods</strong></th>75<th colspan="2"><strong>Overall<sup>Edit</sup>↓</strong></th>76<th colspan="2"><strong>Text<sup>Edit</sup>↓</strong></th>77<th colspan="2"><strong>Formula<sup>Edit</sup>↓</strong></th>78<th colspan="2"><strong>Table<sup>TEDS</sup>↑</strong></th>79<th colspan="2"><strong>Table<sup>Edit</sup>↓</strong></th>80<th colspan="2"><strong>Read Order<sup>Edit</sup>↓</strong></th>81</tr>82<tr>83<th><em>EN</em></th>84<th><em>ZH</em></th>85<th><em>EN</em></th>86<th><em>ZH</em></th>87<th><em>EN</em></th>88<th><em>ZH</em></th>89<th><em>EN</em></th>90<th><em>ZH</em></th>91<th><em>EN</em></th>92<th><em>ZH</em></th>93<th><em>EN</em></th>94<th><em>ZH</em></th>95</tr>96</thead>97<tbody>98<tr>99<td rowspan="8"><strong>Pipeline<br>Tools</strong></td>100<td>MinerU</td>101<td>0.150</td>102<td>0.357</td>103<td>0.061</td>104<td>0.215</td>105<td>0.278</td>106<td>0.577</td>107<td>78.6</td>108<td>62.1</td>109<td>0.180</td>110<td>0.344</td>111<td>0.079</td>112<td>0.292</td>113</tr>114<tr>115<td>Marker</td>116<td>0.336</td>117<td>0.556</td>118<td>0.080</td>119<td>0.315</td>120<td>0.530</td>121<td>0.883</td>122<td>67.6</td>123<td>49.2</td>124<td>0.619</td>125<td>0.685</td>126<td>0.114</td>127<td>0.340</td>128</tr>129<tr>130<td>Mathpix</td>131<td>0.191</td>132<td>0.365</td>133<td>0.105</td>134<td>0.384</td>135<td>0.306</td>136<td>0.454</td>137<td>77.0</td>138<td>67.1</td>139<td>0.243</td>140<td>0.320</td>141<td>0.108</td>142<td>0.304</td>143</tr>144<tr>145<td>Docling</td>146<td>0.589</td>147<td>0.909</td>148<td>0.416</td>149<td>0.987</td>150<td>0.999</td>151<td>1</td>152<td>61.3</td>153<td>25.0</td>154<td>0.627</td>155<td>0.810</td>156<td>0.313</td>157<td>0.837</td>158</tr>159<tr>160<td>Pix2Text</td>161<td>0.320</td>162<td>0.528</td>163<td>0.138</td>164<td>0.356</td>165<td>0.276</td>166<td>0.611</td>167<td>73.6</td>168<td>66.2</td>169<td>0.584</td>170<td>0.645</td>171<td>0.281</td>172<td>0.499</td>173</tr>174<tr>175<td>Unstructured</td>176<td>0.586</td>177<td>0.716</td>178<td>0.198</td>179<td>0.481</td>180<td>0.999</td>181<td>1</td>182<td>0</td>183<td>0.06</td>184<td>1</td>185<td>0.998</td>186<td>0.145</td>187<td>0.387</td>188</tr>189<tr>190<td>OpenParse</td>191<td>0.646</td>192<td>0.814</td>193<td>0.681</td>194<td>0.974</td>195<td>0.996</td>196<td>1</td>197<td>64.8</td>198<td>27.5</td>199<td>0.284</td>200<td>0.639</td>201<td>0.595</td>202<td>0.641</td>203</tr>204<tr>205<td>PPStruct-V3</td>206<td>0.145</td>207<td>0.206</td>208<td>0.058</td>209<td>0.088</td>210<td>0.295</td>211<td>0.535</td>212<td>-</td>213<td>-</td>214<td>0.159</td>215<td>0.109</td>216<td>0.069</td>217<td>0.091</td>218</tr>219<tr>220<td rowspan="9"><strong>Expert<br>VLMs</strong></td>221<td>GOT-OCR</td>222<td>0.287</td>223<td>0.411</td>224<td>0.189</td>225<td>0.315</td>226<td>0.360</td>227<td>0.528</td>228<td>53.2</td>229<td>47.2</td>230<td>0.459</td>231<td>0.520</td>232<td>0.141</td>233<td>0.280</td>234</tr>235<tr>236<td>Nougat</td>237<td>0.452</td>238<td>0.973</td>239<td>0.365</td>240<td>0.998</td>241<td>0.488</td>242<td>0.941</td>243<td>39.9</td>244<td>0</td>245<td>0.572</td>246<td>1.000</td>247<td>0.382</td>248<td>0.954</td>249</tr>250<tr>251<td>Mistral OCR</td>252<td>0.268</td>253<td>0.439</td>254<td>0.072</td>255<td>0.325</td>256<td>0.318</td>257<td>0.495</td>258<td>75.8</td>259<td>63.6</td>260<td>0.600</td>261<td>0.650</td>262<td>0.083</td>263<td>0.284</td>264</tr>265<tr>266<td>OLMOCR-sglang</td>267<td>0.326</td>268<td>0.469</td>269<td>0.097</td>270<td>0.293</td>271<td>0.455</td>272<td>0.655</td>273<td>68.1</td>274<td>61.3</td>275<td>0.608</td>276<td>0.652</td>277<td>0.145</td>278<td>0.277</td>279</tr>280<tr>281<td>SmolDocling-256M</td>282<td>0.493</td>283<td>0.816</td>284<td>0.262</td>285<td>0.838</td>286<td>0.753</td>287<td>0.997</td>288<td>44.9</td>289<td>16.5</td>290<td>0.729</td>291<td>0.907</td>292<td>0.227</td>293<td>0.522</td>294</tr>295<tr>296<td>Dolphin</td>297<td>0.206</td>298<td>0.306</td>299<td>0.107</td>300<td>0.197</td>301<td>0.447</td>302<td>0.580</td>303<td>77.3</td>304<td>67.2</td>305<td>0.180</td>306<td>0.285</td>307<td>0.091</td>308<td>0.162</td>309</tr>310<tr>311<td>MinerU 2</td>312<td>0.139</td>313<td>0.240</td>314<td>0.047</td>315<td>0.109</td>316<td>0.297</td>317<td>0.536</td>318<td>82.5</td>319<td>79.0</td>320<td>0.141</td>321<td>0.195</td>322<td>0.069<</td>323<td>0.118</td>324</tr>325<tr>326<td>OCRFlux</td>327<td>0.195</td>328<td>0.281</td>329<td>0.064</td>330<td>0.183</td>331<td>0.379</td>332<td>0.613</td>333<td>71.6</td>334<td>81.3</td>335<td>0.253</td>336<td>0.139</td>337<td>0.086</td>338<td>0.187</td>339</tr>340<tr>341<td>MonkeyOCR-pro-3B</td>342<td>0.138</td>343<td>0.206</td>344<td>0.067</td>345<td>0.107</td>346<td><strong>0.246</strong></td>347<td>0.421</td>348<td>81.5</td>349<td>87.5</td>350<td>0.139</td>351<td>0.111</td>352<td>0.100</td>353<td>0.185</td>354</tr>355<tr>356 357<td rowspan="5"><strong>General<br>VLMs</strong></td>358<td>GPT4o</td>359<td>0.233</td>360<td>0.399</td>361<td>0.144</td>362<td>0.409</td>363<td>0.425</td>364<td>0.606</td>365<td>72.0</td>366<td>62.9</td>367<td>0.234</td>368<td>0.329</td>369<td>0.128</td>370<td>0.251</td>371</tr>372    <tr>373      <td>Qwen2-VL-72B</td>374      <td>0.252</td>375      <td>0.327</td>376      <td>0.096</td>377      <td>0.218</td>378      <td>0.404</td>379      <td>0.487</td>380      <td>76.8</td>381      <td>76.4</td>382      <td>0.387</td>383      <td>0.408</td>384      <td>0.119</td>385      <td>0.193</td>386    </tr>387    <tr>388      <td>Qwen2.5-VL-72B</td>389      <td>0.214</td>390      <td>0.261</td>391      <td>0.092</td>392      <td>0.18</td>393      <td>0.315</td>394      <td>0.434</td>395      <td>82.9</td>396      <td>83.9</td>397      <td>0.341</td>398      <td>0.262</td>399      <td>0.106</td>400      <td>0.168</td>401    </tr>402    <tr>403      <td>Gemini2.5-Pro</td>404      <td>0.148</td>405      <td>0.212</td>406      <td>0.055</td>407      <td>0.168</td>408      <td>0.356</td>409      <td>0.439</td>410      <td>85.8</td>411      <td>86.4</td>412      <td>0.13</td>413      <td>0.119</td>414      <td>0.049</td>415      <td>0.121</td>416    </tr>417    <tr>418      <td>doubao-1-5-thinking-vision-pro-250428</td>419      <td>0.140</td>420      <td>0.162</td>421      <td>0.043</td>422      <td>0.085</td>423      <td>0.295</td>424      <td><strong>0.384</strong></td>425      <td>83.3</td>426      <td><strong>89.3</strong></td>427      <td>0.165</td>428      <td><strong>0.085</strong></td>429      <td>0.058</td>430      <td>0.094</td>431    </tr>432<tr>433<td rowspan="1"><strong>Expert VLMs</strong></td>434<td><strong>dots.ocr</strong></td>435<td><strong>0.125</strong></td>436<td><strong>0.160</strong></td>437<td><strong>0.032</strong></td>438<td><strong>0.066</strong></td>439<td>0.329</td>440<td>0.416</td>441<td><strong>88.6</strong></td>442<td>89.0</td>443<td><strong>0.099</strong></td>444<td>0.092</td>445<td><strong>0.040</strong></td>446<td><strong>0.067</strong></td>447</tr>448<tr>449</tbody>450</table>451 452 453#### The end-to-end text recognition performance across 9 PDF page types.454 455<table>456<thead>457<tr>458<th><strong>Model<br>Type</strong></th>459<th><strong>Models</strong></th>460<th><strong>Book</strong></th>461<th><strong>Slides</strong></th>462<th><strong>Financial<br>Report</strong></th>463<th><strong>Textbook</strong></th>464<th><strong>Exam<br>Paper</strong></th>465<th><strong>Magazine</strong></th>466<th><strong>Academic<br>Papers</strong></th>467<th><strong>Notes</strong></th>468<th><strong>Newspaper</strong></th>469<th><strong>Overall</strong></th>470</tr>471</thead>472<tbody>473<tr>474<td rowspan="3"><strong>Pipeline<br>Tools</strong></td>475<td>MinerU</td>476<td>0.055</td>477<td>0.124</td>478<td><u>0.033</u></td>479<td>0.102</td>480<td>0.159</td>481<td><strong>0.072</strong></td>482<td><u>0.025</u></td>483<td>0.984</td>484<td>0.171</td>485<td>0.206</td>486</tr>487<tr>488<td>Marker</td>489<td>0.074</td>490<td>0.340</td>491<td>0.089</td>492<td>0.319</td>493<td>0.452</td>494<td>0.153</td>495<td>0.059</td>496<td>0.651</td>497<td>0.192</td>498<td>0.274</td>499</tr>500<tr>501<td>Mathpix</td>502<td>0.131</td>503<td>0.220</td>504<td>0.202</td>505<td>0.216</td>506<td>0.278</td>507<td>0.147</td>508<td>0.091</td>509<td>0.634</td>510<td>0.690</td>511<td>0.300</td>512</tr>513<tr>514<td rowspan="5"><strong>Expert<br>VLMs</strong></td>515<td>GOT-OCR</td>516<td>0.111</td>517<td>0.222</td>518<td>0.067</td>519<td>0.132</td>520<td>0.204</td>521<td>0.198</td>522<td>0.179</td>523<td>0.388</td>524<td>0.771</td>525<td>0.267</td>526</tr>527<tr>528<td>Nougat</td>529<td>0.734</td>530<td>0.958</td>531<td>1.000</td>532<td>0.820</td>533<td>0.930</td>534<td>0.830</td>535<td>0.214</td>536<td>0.991</td>537<td>0.871</td>538<td>0.806</td>539</tr>540<tr>541<td>Dolphin</td>542<td>0.091</td>543<td>0.131</td>544<td>0.057</td>545<td>0.146</td>546<td>0.231</td>547<td>0.121</td>548<td>0.074</td>549<td>0.363</td>550<td>0.307</td>551<td>0.177</td>552</tr>553<tr>554<td>OCRFlux</td>555<td>0.068</td>556<td>0.125</td>557<td>0.092</td>558<td>0.102</td>559<td>0.119</td>560<td>0.083</td>561<td>0.047</td>562<td>0.223</td>563<td>0.536</td>564<td>0.149</td>565</tr>566<tr>567<td>MonkeyOCR-pro-3B</td>568<td>0.084</td>569<td>0.129</td>570<td>0.060</td>571<td>0.090</td>572<td>0.107</td>573<td>0.073</td>574<td>0.050</td>575<td>0.171</td>576<td>0.107</td>577<td>0.100</td>578</tr>579<tr>580<td rowspan="4"><strong>General<br>VLMs</strong></td>581<td>GPT4o</td>582<td>0.157</td>583<td>0.163</td>584<td>0.348</td>585<td>0.187</td>586<td>0.281</td>587<td>0.173</td>588<td>0.146</td>589<td>0.607</td>590<td>0.751</td>591<td>0.316</td>592</tr>593<tr>594<td>Qwen2.5-VL-7B</td>595<td>0.148</td>596<td>0.053</td>597<td>0.111</td>598<td>0.137</td>599<td>0.189</td>600<td>0.117</td>601<td>0.134</td>602<td>0.204</td>603<td>0.706</td>604<td>0.205</td>605</tr>606<tr>607<td>InternVL3-8B</td>608<td>0.163</td>609<td>0.056</td>610<td>0.107</td>611<td>0.109</td>612<td>0.129</td>613<td>0.100</td>614<td>0.159</td>615<td>0.150</td>616<td>0.681</td>617<td>0.188</td>618</tr>619<tr>620<td>doubao-1-5-thinking-vision-pro-250428</td>621<td>0.048</td>622<td>0.048</td>623<td>0.024</td>624<td><strong>0.062</strong></td>625<td>0.085</td>626<td>0.051</td>627<td>0.039</td>628<td><strong>0.096</strong></td>629<td>0.181</td>630<td>0.073</td>631</tr>632<tr>633<td rowspan="1"><strong>Expert VLMs</strong></td>634<td><strong>dots.ocr</strong></td>635<td><strong>0.031</strong></td>636<td><strong>0.047</strong></td>637<td><strong>0.011</strong></td>638<td>0.082</td>639<td><strong>0.079</strong></td>640<td><strong>0.028</strong></td>641<td><strong>0.029</strong></td>642<td>0.109</td>643<td><strong>0.056</strong></td>644<td><strong>0.055</strong></td>645</tr>646 647</tbody>648</table>649 650> **Notes:** 651> - The metrics are from [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR), [OmniDocBench](https://github.com/opendatalab/OmniDocBench), and our own internal evaluations.652> - We delete the Page-header and Page-footer cells in the result markdown.653> - We use tikz_preprocess pipeline to upsample the images to dpi 200.654 655 656### 2. **dots.ocr-bench**657 658This is an inhouse benchmark which contain 1493 pdf images with 100 languages.659 660#### The end-to-end evaluation results of different tasks.661 662<table>663<thead>664<tr>665<th rowspan="1"><strong>Methods</strong></th>666<th colspan="1"><strong>Overall<sup>Edit</sup>↓</strong></th>667<th colspan="1"><strong>Text<sup>Edit</sup>↓</strong></th>668<th colspan="1"><strong>Formula<sup>Edit</sup>↓</strong></th>669<th colspan="1"><strong>Table<sup>TEDS</sup>↑</strong></th>670<th colspan="1"><strong>Table<sup>Edit</sup>↓</strong></th>671<th colspan="1"><strong>Read Order<sup>Edit</sup>↓</strong></th>672</tr>673</thead>674<tbody>675<td>MonkeyOCR-3B</td>676<td>0.483</td>677<td>0.445</td>678<td>0.627</td>679<td>50.93</td>680<td>0.452</td>681<td>0.409</td>682</tr>683<tr>684<td>doubao-1-5-thinking-vision-pro-250428</td>685<td>0.291</td>686<td>0.226</td>687<td>0.440</td>688<td>71.2</td>689<td>0.260</td>690<td>0.238</td>691</tr>692<tr>693<td>doubao-1-6</td>694<td>0.299</td>695<td>0.270</td>696<td>0.417</td>697<td>71.0</td>698<td>0.258</td>699<td>0.253</td>700</tr>701<tr>702<td>Gemini2.5-Pro</td>703<td>0.251</td>704<td>0.163</td>705<td>0.402</td>706<td>77.1</td>707<td>0.236</td>708<td>0.202</td>709</tr>710<tr>711<td><strong>dots.ocr</strong> </td>712<td><strong>0.177</strong></td>713<td><strong>0.075</strong></td>714<td><strong>0.297</strong></td>715<td><strong>79.2</strong></td>716<td><strong>0.186</strong></td>717<td><strong>0.152</strong></td>718</tr>719 720</tbody>721</table>722 723> **Notes:** 724> - We use the same metric calculation pipeline of [OmniDocBench](https://github.com/opendatalab/OmniDocBench).725> - We delete the Page-header and Page-footer cells in the result markdown.726 727#### Layout Detection728 729<table>730<thead>731<tr>732<th rowspan="2"><strong>Method</strong></th>733<th colspan="5" style="text-align: center;"><strong>F1@IoU=.50:.05:.95↑</strong></th>734<th colspan="5" style="text-align: center;"><strong>F1@IoU=.50↑</strong></th>735</tr>736<tr>737<th>Overall</th>738<th>Text</th>739<th>Formula</th>740<th>Table</th>741<th>Picture</th>742<th>Overall</th>743<th>Text</th>744<th>Formula</th>745<th>Table</th>746<th>Picture</th>747</tr>748</thead>749 750<tbody>751<td>DocLayout-YOLO-DocStructBench</td>752<td>0.733</td>753<td>0.694</td>754<td>0.480</td>755<td>0.803</td>756<td>0.619</td>757<td>0.806</td>758<td>0.779</td>759<td>0.620</td>760<td>0.858</td>761<td>0.678</td>762</tr>763 764<tr>765<td>dots.ocr-parse all</td>766<td>0.831</td>767<td>0.801</td>768<td>0.654</td>769<td>0.838</td>770<td>0.748</td>771<td>0.922</td>772<td>0.909</td>773<td>0.770</td>774<td>0.888</td>775<td>0.831</td>776</tr>777 778<tr>779<td> <strong>dots.ocr-detection only</strong> </td>780<td><strong>0.845</strong></td>781<td><strong>0.816</strong></td>782<td><strong>0.716</strong></td>783<td><strong>0.875</strong></td>784<td><strong>0.765</strong></td>785<td><strong>0.930</strong></td>786<td><strong>0.917</strong></td>787<td><strong>0.832</strong></td>788<td><strong>0.918</strong></td>789<td><strong>0.843</strong></td>790</tr>791 792</tbody>793</table>794 795> **Notes:**  796> - prompt_layout_all_en for **parse all**, prompt_layout_only_en for **detection only**, please refer to [prompts](https://github.com/rednote-hilab/dots.ocr/blob/master/dots_ocr/utils/prompts.py)797 798 799### 3. olmOCR-bench.800 801<table>802<thead>803<tr>804<th>Model</th>805<th>ArXiv</th>806<th>Old Scans<br>Math</th>807<th>Tables</th>808<th>Old Scans</th>809<th>Headers and<br>Footers</th>810<th>Multi<br>column</th>811<th>Long Tiny<br>Text</th>812<th>Base</th>813<th>Overall</th>814</tr>815</thead>816<tbody>817<tr>818<td>GOT OCR</td>819<td>52.7</td>820<td>52.0</td>821<td>0.2</td>822<td>22.1</td>823<td>93.6</td>824<td>42.0</td>825<td>29.9</td>826<td>94.0</td>827<td>48.3 ± 1.1</td>828</tr>829<tr>830<td>Marker</td>831<td>76.0</td>832<td>57.9</td>833<td>57.6</td>834<td>27.8</td>835<td>84.9</td>836<td>72.9</td>837<td>84.6</td>838<td>99.1</td>839<td>70.1 ± 1.1</td>840</tr>841<tr>842<td>MinerU</td>843<td>75.4</td>844<td>47.4</td>845<td>60.9</td>846<td>17.3</td>847<td><strong>96.6</strong></td>848<td>59.0</td>849<td>39.1</td>850<td>96.6</td>851<td>61.5 ± 1.1</td>852</tr>853<tr>854<td>Mistral OCR</td>855<td>77.2</td>856<td>67.5</td>857<td>60.6</td>858<td>29.3</td>859<td>93.6</td>860<td>71.3</td>861<td>77.1</td>862<td>99.4</td>863<td>72.0 ± 1.1</td>864</tr>865<tr>866<td>Nanonets OCR</td>867<td>67.0</td>868<td>68.6</td>869<td>77.7</td>870<td>39.5</td>871<td>40.7</td>872<td>69.9</td>873<td>53.4</td>874<td>99.3</td>875<td>64.5 ± 1.1</td>876</tr>877<tr>878<td>GPT-4o<br>(No Anchor)</td>879<td>51.5</td>880<td><strong>75.5</strong></td>881<td>69.1</td>882<td>40.9</td>883<td>94.2</td>884<td>68.9</td>885<td>54.1</td>886<td>96.7</td>887<td>68.9 ± 1.1</td>888</tr>889<tr>890<td>GPT-4o<br>(Anchored)</td>891<td>53.5</td>892<td>74.5</td>893<td>70.0</td>894<td>40.7</td>895<td>93.8</td>896<td>69.3</td>897<td>60.6</td>898<td>96.8</td>899<td>69.9 ± 1.1</td>900</tr>901<tr>902<td>Gemini Flash 2<br>(No Anchor)</td>903<td>32.1</td>904<td>56.3</td>905<td>61.4</td>906<td>27.8</td>907<td>48.0</td>908<td>58.7</td>909<td><strong>84.4</strong></td>910<td>94.0</td>911<td>57.8 ± 1.1</td>912</tr>913<tr>914<td>Gemini Flash 2<br>(Anchored)</td>915<td>54.5</td>916<td>56.1</td>917<td>72.1</td>918<td>34.2</td>919<td>64.7</td>920<td>61.5</td>921<td>71.5</td>922<td>95.6</td>923<td>63.8 ± 1.2</td>924</tr>925<tr>926<td>Qwen 2 VL<br>(No Anchor)</td>927<td>19.7</td>928<td>31.7</td>929<td>24.2</td>930<td>17.1</td>931<td>88.9</td>932<td>8.3</td>933<td>6.8</td>934<td>55.5</td>935<td>31.5 ± 0.9</td>936</tr>937<tr>938<td>Qwen 2.5 VL<br>(No Anchor)</td>939<td>63.1</td>940<td>65.7</td>941<td>67.3</td>942<td>38.6</td>943<td>73.6</td>944<td>68.3</td>945<td>49.1</td>946<td>98.3</td>947<td>65.5 ± 1.2</td>948</tr>949<tr>950<td>olmOCR v0.1.75<br>(No Anchor)</td>951<td>71.5</td>952<td>71.4</td>953<td>71.4</td>954<td><strong>42.8</strong></td>955<td>94.1</td>956<td>77.7</td>957<td>71.0</td>958<td>97.8</td>959<td>74.7 ± 1.1</td>960</tr>961<tr>962<td>olmOCR v0.1.75<br>(Anchored)</td>963<td>74.9</td>964<td>71.2</td>965<td>71.0</td>966<td>42.2</td>967<td>94.5</td>968<td>78.3</td>969<td>73.3</td>970<td>98.3</td>971<td>75.5 ± 1.0</td>972</tr>973<tr>974<td>MonkeyOCR-pro-3B</td>975<td><strong>83.8</strong></td>976<td>68.8</td>977<td>74.6</td>978<td>36.1</td>979<td>91.2</td>980<td>76.6</td>981<td>80.1</td>982<td>95.3</td>983<td>75.8 ± 1.0</td>984</tr>985<tr>986<td><strong>dots.ocr</strong></td>987<td>82.1</td>988<td>64.2</td>989<td><strong>88.3</strong></td>990<td>40.9</td>991<td>94.1</td>992<td><strong>82.4</strong></td>993<td>81.2</td>994<td><strong>99.5</strong></td>995<td><strong>79.1 ± 1.0</strong></td>996</tr>997</tbody>998</table>999 1000 1001> **Note:**1002> - The metrics are from [MonkeyOCR](https://github.com/Yuliang-Liu/MonkeyOCR), 1003[olmocr](https://github.com/allenai/olmocr), and our own internal evaluations.1004> - We delete the Page-header and Page-footer cells in the result markdown.1005 1006 1007 1008# Quick Start1009## 1. Installation1010### Install dots.ocr1011```shell1012conda create -n dots_ocr python=3.121013conda activate dots_ocr1014 1015git clone https://github.com/rednote-hilab/dots.ocr.git1016cd dots.ocr1017 1018# Install pytorch, see https://pytorch.org/get-started/previous-versions/ for your cuda version1019pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 --index-url https://download.pytorch.org/whl/cu1281020pip install -e .1021```1022 1023If you have trouble with the installation, try our [Docker Image](https://hub.docker.com/r/rednotehilab/dots.ocr) for an easier setup, and follow these steps:1024```shell1025git clone https://github.com/rednote-hilab/dots.ocr.git1026cd dots.ocr1027pip install -e .1028```1029 1030 1031### Download Model Weights1032> 💡**Note:** Please use a directory name without periods (e.g., `DotsOCR` instead of `dots.ocr`) for the model save path. This is a temporary workaround pending our integration with Transformers.1033```shell1034python3 tools/download_model.py1035```1036 1037 1038## 2. Deployment1039### vLLM inference1040We highly recommend using vllm for deployment and inference. All of our evaluations results are based on vllm version 0.9.1.1041The [Docker Image](https://hub.docker.com/r/rednotehilab/dots.ocr) is based on the official vllm image. You can also follow [Dockerfile](https://github.com/rednote-hilab/dots.ocr/blob/master/docker/Dockerfile) to build the deployment environment by yourself. 1042 1043```shell1044# You need to register model to vllm at first1045python3 tools/download_model.py1046export hf_model_path=./weights/DotsOCR  # Path to your downloaded model weights, Please use a directory name without periods (e.g., `DotsOCR` instead of `dots.ocr`) for the model save path. This is a temporary workaround pending our integration with Transformers.1047export PYTHONPATH=$(dirname "$hf_model_path"):$PYTHONPATH1048sed -i '/^from vllm\.entrypoints\.cli\.main import main$/a\1049from DotsOCR import modeling_dots_ocr_vllm' `which vllm`  # If you downloaded model weights by yourself, please replace `DotsOCR` by your model saved directory name, and remember to use a directory name without periods (e.g., `DotsOCR` instead of `dots.ocr`) 1050 1051# launch vllm server1052CUDA_VISIBLE_DEVICES=0 vllm serve ${hf_model_path} --tensor-parallel-size 1 --gpu-memory-utilization 0.95  --chat-template-content-format string --served-model-name model --trust-remote-code1053 1054# If you get a ModuleNotFoundError: No module named 'DotsOCR', please check the note above on the saved model directory name.1055 1056# vllm api demo1057python3 ./demo/demo_vllm.py --prompt_mode prompt_layout_all_en1058```1059 1060### Hugginface inference1061```shell1062python3 demo/demo_hf.py1063```1064 1065<details>1066<summary><b>Hugginface inference details</b></summary>1067 1068```python1069import torch1070from transformers import AutoModelForCausalLM, AutoProcessor, AutoTokenizer1071from qwen_vl_utils import process_vision_info1072from dots_ocr.utils import dict_promptmode_to_prompt1073 1074model_path = "./weights/DotsOCR"1075model = AutoModelForCausalLM.from_pretrained(1076    model_path,1077    attn_implementation="flash_attention_2",1078    torch_dtype=torch.bfloat16,1079    device_map="auto",1080    trust_remote_code=True1081)1082processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)1083 1084image_path = "demo/demo_image1.jpg"1085prompt = """Please output the layout information from the PDF image, including each layout element's bbox, its category, and the corresponding text content within the bbox.1086 10871. Bbox format: [x1, y1, x2, y2]1088 10892. Layout Categories: The possible categories are ['Caption', 'Footnote', 'Formula', 'List-item', 'Page-footer', 'Page-header', 'Picture', 'Section-header', 'Table', 'Text', 'Title'].1090 10913. Text Extraction & Formatting Rules:1092    - Picture: For the 'Picture' category, the text field should be omitted.1093    - Formula: Format its text as LaTeX.1094    - Table: Format its text as HTML.1095    - All Others (Text, Title, etc.): Format their text as Markdown.1096 10974. Constraints:1098    - The output text must be the original text from the image, with no translation.1099    - All layout elements must be sorted according to human reading order.1100 11015. Final Output: The entire output must be a single JSON object.1102"""1103 1104messages = [1105        {1106            "role": "user",1107            "content": [1108                {1109                    "type": "image",1110                    "image": image_path1111                },1112                {"type": "text", "text": prompt}1113            ]1114        }1115    ]1116 1117# Preparation for inference1118text = processor.apply_chat_template(1119    messages, 1120    tokenize=False, 1121    add_generation_prompt=True1122)1123image_inputs, video_inputs = process_vision_info(messages)1124inputs = processor(1125    text=[text],1126    images=image_inputs,1127    videos=video_inputs,1128    padding=True,1129    return_tensors="pt",1130)1131 1132inputs = inputs.to("cuda")1133 1134# Inference: Generation of the output1135generated_ids = model.generate(**inputs, max_new_tokens=24000)1136generated_ids_trimmed = [1137    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)1138]1139output_text = processor.batch_decode(1140    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False1141)1142print(output_text)1143 1144```1145 1146</details>1147 1148## 3. Document Parse1149**Based on vLLM server**, you can parse an image or a pdf file using the following commands:1150```bash1151 1152# Parse all layout info, both detection and recognition1153# Parse a single image1154python3 dots_ocr/parser.py demo/demo_image1.jpg1155# Parse a single PDF1156python3 dots_ocr/parser.py demo/demo_pdf1.pdf  --num_threads 64  # try bigger num_threads for pdf with a large number of pages1157 1158# Layout detection only1159python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_layout_only_en1160 1161# Parse text only, except Page-header and Page-footer1162python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_ocr1163 1164# Parse layout info by bbox1165python3 dots_ocr/parser.py demo/demo_image1.jpg --prompt prompt_grounding_ocr --bbox 163 241 1536 7051166 1167```1168 1169<details>1170<summary><b>Output Results</b></summary>1171 11721.  **Structured Layout Data** (`demo_image1.json`): A JSON file containing the detected layout elements, including their bounding boxes, categories, and extracted text.11732.  **Processed Markdown File** (`demo_image1.md`): A Markdown file generated from the concatenated text of all detected cells.1174    *   An additional version, `demo_image1_nohf.md`, is also provided, which excludes page headers and footers for compatibility with benchmarks like Omnidocbench and olmOCR-bench.11753.  **Layout Visualization** (`demo_image1.jpg`): The original image with the detected layout bounding boxes drawn on it.1176 1177</details>1178 1179## 4. Demo1180You can run the demo with the following command, or try directly at [live demo](https://dotsocr.xiaohongshu.com/)1181```bash1182python demo/demo_gradio.py1183```1184 1185We also provide a demo for grounding ocr:1186```bash1187python demo/demo_gradio_annotion.py1188```1189 1190 1191### Example for formula document1192<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/showcase/formula1.png" alt="formula1.png" border="0" />1193<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/showcase/formula2.png" alt="formula2.png" border="0" />1194<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/showcase/formula3.png" alt="formula3.png" border="0" />1195 1196### Example for table document1197<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/showcase/table1.png" alt="table1.png" border="0" />1198<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/showcase/table2.png" alt="table2.png" border="0" />1199<img src="https://raw.githubusercontent.com/rednote-hilab/dots.ocr/master/assets/showcase/table3.png" alt="table3.png" border="0" />1200 

Showing the first 1,200 of 1235 lines. Download the file for the rest.