numind/NuExtract3
36560k
1---2license: apache-2.03license_link: https://huggingface.co/numind/NuExtract3/blob/main/LICENSE4library_name: transformers5pipeline_tag: image-to-text6tags:7- image-text-to-text8- transformers9- safetensors10- qwen3_511- vision-language12- vlm13- document-understanding14- structured-extraction15- information-extraction16- ocr17- document-to-markdown18- markdown19- rag20- reasoning21- multilingual22- conversational23base_model:24- Qwen/Qwen3.5-4B25model_name: NuExtract326---27 28<p align="center">29 <a href="https://nuextract.ai/">30 <img src="header.svg" width="900px"/>31 </a>32</p>33 34 35<p align="center">36 🖥️ <a href="https://nuextract.ai/">API / Platform</a> | 37 📑 <a href="https://numind.ai/blog">Blog</a> | 38 🗣️ <a href="https://discord.gg/3tsEtJNCDe">Discord</a> | 39 🛠️ <a href="https://github.com/numindai/nuextract">GitHub</a>40</p>41 42**NuExtract3** is a unified **4B** vision-language reasoning model for document understanding.43 44It combines strong **structured information extraction** with high-quality **image-to-Markdown** conversion, making it suitable for extraction pipelines, OCR, and RAG preprocessing for all types of documents such as scans, receipts, forms, invoices, contracts or tables.45 46Try it out in [the 🤗 space!](https://huggingface.co/spaces/numind/NuExtract-3-4B)47 48## Overview49 50- **Structured extraction**: input (text/images) + JSON template + instructions --> JSON output51- **Markdown conversion**: input (text/images) --> Markdown52- **Multimodal inputs**: text, images, or text + images.53- **Multilingual** documents.54- **Reasoning** and non-reasoning inference modes.55- **Template generation** for structured extraction from natural language or input document.56 57# Benchmark results58 59## Structured Extraction60 61We benchmarked NuExtract on NuMind's internal structured benchmark, measuring model's performances on ~600 documents of diverse types including invoices, movie posters or floor plans. These documents and their ground-truth cover diverse use-cases testing model visual understanding, OCR, reasoning and ability to handle long input and output contexts.62We plan to open-source this benchmark in the coming weeks, along with a extensive leaderboard including most popular open-weight and closed-sourced APIs and a Python library allowing to easily measure model performances on structured extraction.63 64<img src="st.svg" width="1000"/>65 66To measure a pair of predicted and ground-truth JSONs, we represent both as trees which we align based on node names, compute metric scores for aligned leaves and report the average of these scores. `string` and `verbatim-string` leaves are evaluated with indel distance (i.e. Levenshtein without replacement), while all others are evaluated with exact-match.67Models were evaluated using vllm, with a temperature of 0.25 and a maximum of 65000 output token (for both thinking and answer), which largely exceeds 22000 which is the number of tokens of the largest ground truth output.68 69<figure>70 71|Model name |Average score|Num. failed⁽¹⁾|Avg. num tokens thinking|Avg. num tokens answer|72|--------------------|-------------|-----------|------------------------|----------------------|73|NuExtract3.4_4B-RL |**0.651 ± 0.019**|27 |2036 |1856 |74|gemma-4-E4B-it |0.538 ± 0.023|31 |3005 |1287 |75|Qwen3.5-9B |0.479 ± 0.030|170 |22409 |1257 |76|Qwen3.5-4B |0.417 ± 0.031|229 |27177 |1201 |77|GLM-4.6V-Flash |0.435 ± 0.026|153 |2989 |1357 |78|Nemotron-3-Nano-Omni|0.387 ± 0.028|204 |25827 |522 |79|Ministral-3-3B |0.240 ± 0.022|344 |27586 |362 |80 81<figcaption>82<small>83(1) number of model outputs that were not JSON deserializable, either directly or by removing leading and trailing backticks.<br>8495% confidence intervals computed using a nonparametric bootstrap over scores distributions.85</small>86</figcaption>87</figure>88 89The benchmark include samples containing multiple images resulting in large input context, and some with ground-truth containing large numbers of items to extract resulting in large outputs. We found that the reasoning of small models significantly negatively impact their performances. The reason is that many models ended up falling in repetition loops, hitting the output tokens limit and resulting in failed requests.90 91## Document to Markdown92 93NuExtract can also convert document images into clean Markdown. Output will be Markdown for text (headers etc), HTML for tables, LaTeX for math and ```<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> ```94 95Modern, format-agnostic benchmarks for complex document understanding are limited, so we explored a new evaluation approach.96We selected 100 documents with challenging layouts and tables, asked each model to convert them into a structured representation, then used Gemini 3 Flash to compare model outputs against the source document and choose the most accurate result.97The rankings aligned with human votes, suggesting this is a promising method for evaluating document-to-Markdown capabilities. More details will be shared in an upcoming technical report.98Here are some results:99 100<img src="ocr_preferences.svg" width="1000"/>101 102### Using "Markdown-to-structured"103 104To add other evaluate references, we used our structured extraction benchmark to evaluate models in a two-step fashion: convert the benchmark inputs to Markdown, then use Qwen3.6 27B to perform the structured extraction task on them. Intuitively, it allows to evaluate how models achieve to keep the input document content and layout: good models will allow the "structured extractor" model to perform better scores.105 106<img src="md2st.svg" width="1000"/>107 108 109# Using NuExtract110 111## Structured extraction112 113Structured extraction takes as inputs:114 1151. An input document, which can be text, image, or both;1162. A JSON template describing the information to extract;1173. (Optional) Instructions, allowing to specify expected output formats or values, to provide with the `instructions` chat template kwarg;1184. (Optional) In-Context Learning (ICL) examples.119 120### Input JSON template121 122NuExtract uses a input JSON template whose structure is identical to the output JSON. Its leaf values are specify the **types** of the output JSON leaves. For examples:123 124```json125{126 "invoice_number": "verbatim-string",127 "invoice_date": "date",128 "total_amount": "number",129 "currency": "currency",130 "line_items": [131 {132 "description": "verbatim-string",133 "item_type": ["electronics", "clothing", "vehicle", "furniture", "other"],134 "quantity": "integer",135 "unit_price": "number",136 "total": "number"137 }138 ]139}140```141 142Supported template types include:143 144- `verbatim-string`: extract text exactly as it appears in the document;145- `string`: generic string field, allowing abstraction or light paraphrasing;146- `integer`: whole number;147- `number`: integer or decimal number;148- `date-time`: ISO-8601 date, time or date-time;149- Other specific types such as `data`, `time`, `country`, `currency`, `email` and so on.150[**For more details, read the complete types specifications and examples**](TYPES.md)151 152Template constructors:153 154- Arrays, for example `["string"]`;155- Enums, for example `["yes", "no", "maybe"]`;156- Multi-enums (multiple possible values), for example `[["A", "B", "C"]]`.157 158If the model does not find relevant information for a field, it returns `null` or `[]`.159 160### Converting JSON schema / Pydantic models to NuExtract template161 162Our Python SDK (`pip install numind`) offers a method to convert JSON schemas to NuExtract templates:163 164```Python165from typing import Literal166 167from pydantic import Field, BaseModel168from numind.nuextract_utils import convert_json_schema_to_nuextract_template169 170 171class HotelBooking(BaseModel):172 city: str173 check_in_date: str = Field(description="date")174 check_out_date: str = Field(description="date")175 number_of_guests: int176 room_type: Literal["single", "double", "suite"]177 178 179template, dropped_branches = convert_json_schema_to_nuextract_template(180 HotelBooking.model_json_schema()181)182 183# {'check_in_date': 'date', 'check_out_date': 'date', 'city': 'string', 'number_of_guests': 'integer', 'room_type': ['single', 'double', 'suite']}184```185 186## Document-to-Markdown187 188NuExtract can also convert document images into clean Markdown. Output will be markdown for text (headers etc), html for tables, latex for mat and ```<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/> ```189 190Markdown example:191 192```markdown193<figure data-type="image" data-id="img_1">194 <img src="img_1.png" alt="Logo of Mobilier 2000 with contact information: Tél.: (418) 275-4232, 1654, boul. Marcotte, Roberval (Qc) G8H 2P2"/>195</figure>196 197# COMMANDE198**NUMÉRO 72259**199 2001201 202**Vendu à**203TREMBLAY ERIC204ERIC TREMBLAY205348 BOUL. DE L'ANSE206ROBERVAL207G8H 1Y9208 209**Livré à**210TREMBLAY ERIC211ERIC TREMBLAY212348 BOUL. DE L'ANSE213ROBERVAL214G8H 1Y9215 216<table>217 <thead>218 <tr>219 <th># CLIENT</th>220 <th>EXPÉDITEUR</th>221 <th>TERME DE CRÉDIT</th>222 <th>DATE</th>223 </tr>224 </thead>225 <tbody>226 <tr>227 <td>2753133</td>228 <td>Notre camion</td>229 <td>à la livraison</td>230 <td>22/06/2023</td>231 </tr>232 </tbody>233</table>234 235<table>236 <thead>237 <tr>238 <th>NOM DU VENDEUR</th>239 <th>VOTRE ÉCONOMIE !</th>240 <th># COMMANDE</th>241 </tr>242 </thead>243 <tbody>244 <tr>245 <td>Éric</td>246 <td>0.00</td>247 <td></td>248 </tr>249 </tbody>250</table>251```252 253---254 255## Reasoning and non-reasoning modes256 257NuExtract supports both reasoning and non-reasoning inference.258 259### Non-thinking mode260 261Use this for fast and deterministic extraction or Markdown conversion.262 263```python264enable_thinking = False265temperature = 0.2266```267 268### Thinking mode269 270Use this for difficult documents, complex layouts, ambiguous fields, or cases where the document structure requires additional reasoning.271 272```python273enable_thinking = True274temperature = 0.6275```276 277For production extraction workloads, we recommend starting with **non-reasoning mode** and enabling reasoning only for difficult examples.278 279 280---281 282## vLLM deployment283 284NuExtract can be served with vLLM using an OpenAI-compatible API.285 286```bash287vllm serve numind/NuExtract3 \288 --trust-remote-code \289 --limit-mm-per-prompt '{"image": 99, "video": 0}' \290 --chat-template-content-format openai \291 --generation-config vllm \292 --max-model-len 131072 \293 --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'294```295 296 297### Multi Token Prediction298<details>299The deployment commands above enable Multi Token Prediction (MTP) through vLLM speculative decoding:300 301```bash302--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'303```304 305MTP can improve decoding throughput without changing the OpenAI-compatible request payload. You can tune `num_speculative_tokens` for your hardware and workload, or remove `--speculative-config` if your vLLM version or environment does not support this speculative decoding method.306 307If you encounter memory issues, reduce the maximum model length and the maximum number of images:308 309```bash310vllm serve numind/NuExtract-3 \311 --trust-remote-code \312 --limit-mm-per-prompt '{"image": 6, "video": 0}' \313 --chat-template-content-format openai \314 --generation-config vllm \315 --max-model-len 16384 \316 --speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}'317```318</details>319 320## vLLM inference: structured extraction: text321```python322import json323from openai import OpenAI324 325client = OpenAI(326 api_key="EMPTY",327 base_url="http://localhost:8000/v1",328)329 330template = {331 "store": "verbatim-string",332 "date": "date-time",333 "total": "number",334 "currency": ["USD", "EUR", "GBP", "JPY", "Other"],335 "items": [336 {337 "name": "verbatim-string",338 "price": "number"339 }340 ]341}342 343response = client.chat.completions.create(344 model="numind/NuExtract3",345 temperature=0.2,346 messages=[347 {348 "role": "user",349 "content": [350 {351 "type": "text",352 "text": "Yesterday I bought apples and coffee at Trader Joe's for a total of $12.40."353 }354 ],355 }356 ],357 extra_body={358 "chat_template_kwargs": {359 "template": json.dumps(template),360 "instructions": "Specify the time for the `date` entry only if it is present, otherwise only output the date component.",361 "enable_thinking": False362 }363 }364)365 366print(response.choices[0].message.content)367```368 369Example output:370 371```json372{373 "store": "Trader Joe's",374 "date": null,375 "total": 12.40,376 "currency": "USD",377 "items": [378 {379 "name": "apples",380 "price": null381 },382 {383 "name": "coffee",384 "price": null385 }386 ]387}388```389 390---391 392## vLLM inference: structured extraction: image393 394```python395import json396import base64397from openai import OpenAI398 399client = OpenAI(400 api_key="EMPTY",401 base_url="http://localhost:8000/v1",402)403 404def encode_image(image_path):405 with open(image_path, "rb") as image_file:406 return base64.b64encode(image_file.read()).decode("utf-8")407 408image_base64 = encode_image("receipt.png")409data_url = f"data:image/png;base64,{image_base64}"410 411template = {412 "store": "verbatim-string",413 "date": "date-time",414 "total": "number",415 "payment_method": "verbatim-string"416}417 418response = client.chat.completions.create(419 model="numind/NuExtract3",420 temperature=0.2,421 messages=[422 {423 "role": "user",424 "content": [425 {426 "type": "image_url",427 "image_url": {"url": data_url}428 }429 ],430 }431 ],432 extra_body={433 "chat_template_kwargs": {434 "template": json.dumps(template, indent=4),435 "enable_thinking": False436 }437 }438)439 440print(response.choices[0].message.content)441```442 443Example output:444 445```json446{447 "store": "Trader Joe's",448 "date": "2025-04-12",449 "total": 42.85,450 "payment_method": "Visa"451}452```453 454### Multiple page PDF455<details>456You can render a PDF to one PNG image per page with PyMuPDF, then pass the images to vLLM in page order.457 458```python459import base64460import json461 462import fitz # pip install pymupdf463from openai import OpenAI464 465client = OpenAI(466 api_key="EMPTY",467 base_url="http://localhost:8000/v1",468)469 470def pdf_to_png_data_urls(pdf_path, dpi=170):471 data_urls = []472 473 with fitz.open(pdf_path) as doc:474 for page in doc:475 pix = page.get_pixmap(dpi=dpi, alpha=False)476 png_bytes = pix.tobytes("png")477 png_base64 = base64.b64encode(png_bytes).decode("utf-8")478 data_urls.append(f"data:image/png;base64,{png_base64}")479 480 return data_urls481 482data_urls = pdf_to_png_data_urls("invoice.pdf", dpi=170)483 484template = {485 "invoice_number": "verbatim-string",486 "invoice_date": "date",487 "total": "number",488 "currency": "currency",489 "line_items": [490 {491 "description": "verbatim-string",492 "quantity": "number",493 "unit_price": "number",494 "total": "number"495 }496 ]497}498 499response = client.chat.completions.create(500 model="numind/NuExtract3",501 temperature=0.2,502 messages=[503 {504 "role": "user",505 "content": [506 {507 "type": "image_url",508 "image_url": {"url": data_url}509 }510 for data_url in data_urls511 ],512 }513 ],514 extra_body={515 "chat_template_kwargs": {516 "template": json.dumps(template, indent=4),517 "enable_thinking": False518 }519 }520)521 522print(response.choices[0].message.content)523```524</details>525 526 527 528## vLLM inference: document-to-Markdown529 530For Markdown OCR, use `mode="markdown"` or `mode="content"` without a template.531 532```python533import base64534from openai import OpenAI535 536client = OpenAI(537 api_key="EMPTY",538 base_url="http://localhost:8000/v1",539)540 541def encode_image(image_path):542 with open(image_path, "rb") as image_file:543 return base64.b64encode(image_file.read()).decode("utf-8")544 545image_base64 = encode_image("document.png")546data_url = f"data:image/png;base64,{image_base64}"547 548response = client.chat.completions.create(549 model="numind/NuExtract3",550 temperature=1,551 messages=[552 {553 "role": "user",554 "content": [555 {556 "type": "image_url",557 "image_url": {"url": data_url}558 }559 ],560 }561 ],562 extra_body={563 "chat_template_kwargs": {564 "mode": "markdown",565 "enable_thinking": False566 }567 }568)569 570print(response.choices[0].message.content)571```572 573---574 575## vLLM inference: reasoning mode576<details>577Reasoning can be enabled for harder structured extraction or Markdown tasks.578 579```python580response = client.chat.completions.create(581 model="numind/NuExtract3",582 temperature=1,583 messages=[584 {585 "role": "user",586 "content": [587 {588 "type": "image_url",589 "image_url": {"url": data_url}590 }591 ],592 }593 ],594 extra_body={595 "chat_template_kwargs": {596 "mode": "markdown",597 "enable_thinking": True598 }599 }600)601 602result = response.choices[0].message.content603 604if "</think>" in result:605 reasoning, answer = [part.strip() for part in result.split("</think>")]606else:607 reasoning, answer = None, result608 609print(answer)610```611</details>612 613 614## In-context examples for extraction615<details>616NuExtract supports in-context examples for structured extraction.617 618Examples are especially useful when the desired formatting is ambiguous or when the schema requires task-specific conventions. Examples can be provided by using `developer` messages, for which all items of the contents except the last one are the input, and the last one is the expected output.619 620```python621import json622from openai import OpenAI623 624client = OpenAI(625 api_key="EMPTY",626 base_url="http://localhost:8000/v1",627)628 629template = {630 "names": ["string"]631}632 633response = client.chat.completions.create(634 model="numind/NuExtract3",635 temperature=0.2,636 messages=[637 {638 "role": "developer",639 "content": [640 {641 "type": "text",642 "text": "Stephen is the manager at Susan's store.",643 },644 {645 "type": "text",646 "text": "{\"names\": [\"-STEPHEN-\", \"-SUSAN-\"]}",647 }648 ],649 },650 {651 "role": "user",652 "content": [653 {654 "type": "text",655 "text": "John went to the restaurant with Mary. James went to the cinema."656 }657 ],658 }659 ],660 extra_body={661 "chat_template_kwargs": {662 "template": json.dumps(template, indent=4),663 "enable_thinking": False664 }665 }666)667 668print(response.choices[0].message.content)669```670 671Example output:672 673```json674{675 "names": ["-JOHN-", "-MARY-", "-JAMES-"]676}677```678</details>679 680 681## vLLM inference: template generation682 683NuExtract can generate an extraction template from a natural language description.684 685```python686from openai import OpenAI687 688client = OpenAI(689 api_key="EMPTY",690 base_url="http://localhost:8000/v1",691)692 693response = client.chat.completions.create(694 model="numind/NuExtract3",695 temperature=0.2,696 messages=[697 {698 "role": "user",699 "content": [700 {701 "type": "text",702 "text": "I want to extract the key details from a rental contract."703 }704 ],705 }706 ],707 extra_body={708 "chat_template_kwargs": {709 "mode": "template-generation"710 }711 }712)713 714print(response.choices[0].message.content)715```716 717Example output:718 719```json720{721 "contract_title": "verbatim-string",722 "landlord": "verbatim-string",723 "tenant": "verbatim-string",724 "property_address": "verbatim-string",725 "start_date": "date-time",726 "end_date": "date-time",727 "monthly_rent": "number",728 "currency": "verbatim-string",729 "deposit": "number",730 "signatories": ["verbatim-string"]731}732```733 734## Curl examples735<details>736 737The following examples assume that vLLM is running locally on port 8000. They use `jq` to build valid JSON request bodies without manually escaping the image data or template string.738 739### Single image structured extraction740 741```bash742API_KEY="EMPTY"743IMAGE_BASE64_FILE=$(mktemp)744REQUEST_BODY_FILE=$(mktemp)745 746base64 < receipt.png | tr -d '\n' > "$IMAGE_BASE64_FILE"747 748TEMPLATE=$(cat <<'JSON'749{750 "store": "verbatim-string",751 "date": "date-time",752 "total": "number",753 "payment_method": "verbatim-string"754}755JSON756)757 758jq -n \759 --rawfile image_base64 "$IMAGE_BASE64_FILE" \760 --arg template "$TEMPLATE" \761 '{762 model: "numind/NuExtract3",763 temperature: 0.6,764 messages: [765 {766 role: "user",767 content: [768 {769 type: "image_url",770 image_url: {url: ("data:image/png;base64," + $image_base64)}771 }772 ]773 }774 ],775 chat_template_kwargs: {776 template: $template,777 enable_thinking: false778 }779 }' > "$REQUEST_BODY_FILE"780 781curl http://localhost:8000/v1/chat/completions \782 -H "Content-Type: application/json" \783 -H "Authorization: Bearer $API_KEY" \784 --data-binary "@$REQUEST_BODY_FILE"785 786rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"787```788 789### Single image content extraction790 791```bash792API_KEY="EMPTY"793IMAGE_BASE64_FILE=$(mktemp)794REQUEST_BODY_FILE=$(mktemp)795 796base64 < document.png | tr -d '\n' > "$IMAGE_BASE64_FILE"797 798jq -n \799 --rawfile image_base64 "$IMAGE_BASE64_FILE" \800 '{801 model: "numind/NuExtract3",802 temperature: 0.6,803 messages: [804 {805 role: "user",806 content: [807 {808 type: "image_url",809 image_url: {url: ("data:image/png;base64," + $image_base64)}810 }811 ]812 }813 ],814 chat_template_kwargs: {815 mode: "content",816 enable_thinking: false817 }818 }' > "$REQUEST_BODY_FILE"819 820curl http://localhost:8000/v1/chat/completions \821 -H "Content-Type: application/json" \822 -H "Authorization: Bearer $API_KEY" \823 --data-binary "@$REQUEST_BODY_FILE"824 825rm "$IMAGE_BASE64_FILE" "$REQUEST_BODY_FILE"826```827</details>828 829 830## Transformers example831<details>832You can also run NuExtract directly with `transformers`. The same `template`, `mode`, and `enable_thinking` options are passed to `processor.apply_chat_template`.833 834```python835import json836 837import torch838from PIL import Image839from transformers import AutoModelForImageTextToText, AutoProcessor840 841model_id = "numind/NuExtract3"842 843processor = AutoProcessor.from_pretrained(844 model_id,845 trust_remote_code=True,846)847model = AutoModelForImageTextToText.from_pretrained(848 model_id,849 dtype=torch.bfloat16,850 device_map="auto",851 trust_remote_code=True,852).eval()853 854def run_nuextract(messages, **chat_template_kwargs):855 inputs = processor.apply_chat_template(856 messages,857 add_generation_prompt=True,858 tokenize=True,859 return_dict=True,860 return_tensors="pt",861 **chat_template_kwargs,862 ).to(model.device)863 864 with torch.inference_mode():865 generated_ids = model.generate(866 **inputs,867 max_new_tokens=4096,868 do_sample=False,869 )870 871 generated_ids = generated_ids[:, inputs.input_ids.shape[1]:]872 return processor.batch_decode(873 generated_ids,874 skip_special_tokens=True,875 clean_up_tokenization_spaces=False,876 )[0].strip()877 878# Single image structured extraction879receipt_image = Image.open("receipt.png").convert("RGB")880receipt_messages = [881 {882 "role": "user",883 "content": [884 {885 "type": "image",886 "image": receipt_image,887 }888 ],889 }890]891 892template = {893 "store": "verbatim-string",894 "date": "date-time",895 "total": "number",896 "payment_method": "verbatim-string"897}898 899structured_output = run_nuextract(900 receipt_messages,901 template=json.dumps(template, indent=4),902 enable_thinking=False,903)904print(structured_output)905 906# Single image content extraction907document_image = Image.open("document.png").convert("RGB")908document_messages = [909 {910 "role": "user",911 "content": [912 {913 "type": "image",914 "image": document_image,915 }916 ],917 }918]919 920content_output = run_nuextract(921 document_messages,922 mode="content",923 enable_thinking=False,924)925print(content_output)926```927</details>928 929Special thanks to the Lambda.ai team for the compute that made this project a success.930 931## Citation932 933If you use NuExtract, please cite NuMind and link to the model page.934 935```bibtex936@misc{nuextract3,937 title = {NuExtract3},938 author = {NuMind},939 year = {2026},940 url = {https://nuextract.ai/}941}942```