ricklon/DeepSeek-OCR-2-Math

1# Technical Documentation2 3This document covers the implementation details of the DeepSeek-OCR-2 Math Rendering Edition. It is intended for developers who want to understand, extend, or debug the pipeline.4 5---6 7## Table of Contents8 91. [Architecture Overview](#architecture-overview)100. [VRAM Usage and Quantized Models](#vram-usage-and-quantized-models)112. [Prompts and Special Tokens](#prompts-and-special-tokens)123. [Grounding and Layout Detection](#grounding-and-layout-detection)134. [Figure and Graph Extraction](#figure-and-graph-extraction)145. [stdout Capture Pattern](#stdout-capture-pattern)156. [PDF Rendering](#pdf-rendering) — image conversion, 300 DPI rationale, one page at a time, digital vs scanned167. [Dual-pass Output Cleaning](#dual-pass-output-cleaning)178. [Bounding Box Rendering](#bounding-box-rendering)189. [Math Rendering Pipeline](#math-rendering-pipeline)1910. [Known Quirks and Workarounds](#known-quirks-and-workarounds)20 21---22 23## Architecture Overview24 25```26User input (image or PDF)27        │28        ▼29  PDF? ─── fitz renders page at 300 DPI ──► PIL Image30  No?  ─── PIL Image directly31        │32        ▼33  model.infer() called with prompt + image path34        │35        ▼ (stdout captured)36  Raw model output (text + grounding tokens)37        │38        ├──► clean_output(include_images=False) ──► Text tab39        │40        ├──► clean_output(include_images=True)41        │           │42        │           ▼43        │    embed_images() ──► Markdown string with base64 figures44        │           │45        │           ▼46        │    to_math_html() ──► HTML with MathJax ──► Markdown Preview tab47        │48        ├──► extract_grounding_references()49        │           │50        │           ▼51        │    draw_bounding_boxes() ──► Boxes tab52        │                         └──► crops ──► Cropped Images tab53        │54        └──► raw result ──► Raw Text tab55```56 57---58 59## Prompts and Special Tokens60 61Each task sends a different prompt to the model. The prompt controls both what the model outputs and whether it performs layout detection.62 63| Task | Prompt | Grounding |64|---|---|---|65| Markdown | `<image>\n<\|grounding\|>Convert the document to markdown.` | Yes |66| Free OCR | `<image>\nFree OCR.` | No |67| Locate | `<image>\nLocate <\|ref\|>text<\|/ref\|> in the image.` | Yes |68| Describe | `<image>\nDescribe this image in detail.` | No |69| Custom | User-defined | Optional |70 71### Special tokens72 73| Token | Purpose |74|---|---|75| `<image>` | Replaced at inference time with visual patch embeddings from the input image |76| `<\|grounding\|>` | Activates layout detection mode — the model annotates every detected region with a label and bounding box |77| `<\|ref\|>label<\|/ref\|>` | Wraps the label of a detected region (e.g. `title`, `text`, `image`, `table`) |78| `<\|det\|>coords<\|/det\|>` | Wraps the bounding box coordinates for that region |79 80### Locate task81 82When using Locate, the user's input is embedded directly into the prompt:83 84```python85prompt = f"<image>\nLocate <|ref|>{custom_prompt.strip()}<|/ref|> in the image."86```87 88This asks the model to find a specific string or element and return its bounding box coordinates.89 90---91 92## Grounding and Layout Detection93 94When `<|grounding|>` is present, the model interleaves its text output with region annotations. A typical raw output looks like:95 96```97# Introduction98<|ref|>title<|/ref|><|det|>[[45, 12, 820, 48]]<|/det|>99 100This paper presents a method for...101<|ref|>text<|/ref|><|det|>[[45, 60, 820, 340]]<|/det|>102 103<|ref|>image<|/ref|><|det|>[[45, 360, 820, 680]]<|/det|>104 105| A | B |106|---|---|107<|ref|>table<|/ref|><|det|>[[45, 700, 820, 900]]<|/det|>108```109 110The labels (`title`, `text`, `image`, `table`) are part of the model's training vocabulary — the model assigns them based on what it detects, not from any hardcoded list in the app.111 112The regex that parses this is:113 114```python115pattern = r'(<\|ref\|>(.*?)<\|/ref\|><\|det\|>(.*?)<\|/det\|>)'116```117 118This returns a list of tuples: `(full_match, label, coordinates_string)`.119 120### Coordinate system121 122Bounding box coordinates are normalised to a **0–999 scale**, not pixel coordinates. The app scales them back at render time:123 124```python125x1 = int(box[0] / 999 * img_w)126y1 = int(box[1] / 999 * img_h)127```128 129This means coordinates are resolution-independent — the same model output works regardless of the original image size.130 131---132 133## Figure and Graph Extraction134 135Graph and figure extraction is a side effect of bounding box processing. Inside `draw_bounding_boxes()`:136 137```python138if extract_images and label == 'image':139    crops.append(image.crop((x1, y1, x2, y2)))140```141 142Only regions the model labels as `'image'` are cropped. Text blocks, titles, and tables get bounding boxes drawn but are not extracted.143 144These crops are then:1451. Added to the **Cropped Images** gallery tab1462. Base64-encoded and embedded into the markdown as `![Figure N](data:image/png;base64,...)` by `embed_images()`, so they appear inline in the **Markdown Preview** tab147 148---149 150## stdout Capture Pattern151 152The model's `infer()` method was designed as a CLI tool — it `print()`s its output rather than returning it. The app captures this by temporarily replacing `sys.stdout`:153 154```python155stdout = sys.stdout156sys.stdout = StringIO()157 158model.infer(...)159 160raw = sys.stdout.getvalue()161sys.stdout = stdout162```163 164The model also prints internal diagnostics alongside the actual output. These are filtered out by checking for known debug strings:165 166```python167debug_filters = ['PATCHES', '====', 'BASE:', 'directly resize',168                 'NO PATCHES', 'torch.Size', '%|']169 170result = '\n'.join([171    l for l in raw.split('\n')172    if l.strip() and not any(s in l for s in debug_filters)173])174```175 176If inference ever produces unexpected empty output, checking what the model is printing to stdout (by temporarily removing the capture) is the first debugging step.177 178---179 180## PDF Rendering181 182### PDFs are converted to images — the text layer is never read183 184The app does not extract embedded text from PDFs. Every page is rasterised to a PNG image first, then passed through the exact same pipeline as a directly uploaded image:185 186```python187def process_pdf(path, task, custom_prompt, page_num):188    doc = fitz.open(path)189    page = doc.load_page(page_num - 1)190    pix = page.get_pixmap(matrix=fitz.Matrix(300/72, 300/72), alpha=False)191    img = Image.open(BytesIO(pix.tobytes("png")))192    doc.close()193    return process_image(img, task, custom_prompt)194```195 196This means the model reads pixels, not characters. It has no access to the PDF's internal text layer, font metadata, or document structure.197 198### Why 300 DPI199 200PDFs are natively specified at 72 DPI. The `fitz.Matrix(300/72, 300/72)` call scales the render up by ~4.17×:201 202- At 72 DPI, small text, subscripts, superscripts, and fine math symbols are too coarse for the model to read reliably203- At 300 DPI, characters are sharp enough for accurate OCR even at small point sizes204- 300 DPI is the standard used by document scanners for archival quality205 206### One page at a time207 208The current implementation processes one page per submission. There is no batch mode. For a multi-page document the user selects a page number, submits, then moves to the next page.209 210The page selector UI is only shown when a PDF is uploaded:211 212```python213def update_page_selector(file_path):214    if file_path.lower().endswith('.pdf'):215        page_count = get_pdf_page_count(file_path)216        return gr.update(visible=True, maximum=page_count, value=1, minimum=1)217    return gr.update(visible=False)218```219 220### Digital vs scanned PDFs221 222Both work identically:223 224| PDF type | What's inside | Result |225|---|---|---|226| Digital (text-based) | Vector fonts and geometry | PyMuPDF re-rasterises from vectors — output is perfectly sharp at any DPI |227| Scanned | Embedded raster images | PyMuPDF extracts the raster — output quality depends on the original scan resolution |228 229For scanned PDFs with low source resolution (e.g. 150 DPI originals), upscaling to 300 DPI will not recover detail that was never there. In those cases inference accuracy may be lower than with high-quality digital PDFs.230 231---232 233## Dual-pass Output Cleaning234 235`clean_output()` is called twice on the same raw result to produce two different outputs:236 237```python238cleaned  = clean_output(result, include_images=False)  # → Text tab239markdown = clean_output(result, include_images=True)   # → Markdown Preview240```241 242With `include_images=False`:243- Grounding tokens are stripped244- `<|ref|>image<|/ref|>` regions are removed entirely245- Result is clean plain text246 247With `include_images=True`:248- Text grounding tokens are stripped249- `<|ref|>image<|/ref|>` regions are replaced with `**[Figure N]**` placeholders250- `embed_images()` then swaps those placeholders for actual base64-encoded PNGs251 252---253 254## Bounding Box Rendering255 256Bounding boxes are drawn in two layers using Pillow:257 2581. **Solid outline** — drawn directly on a copy of the image2592. **Semi-transparent fill** — drawn on a separate RGBA overlay, then composited260 261```python262overlay = Image.new('RGBA', img_draw.size, (0, 0, 0, 0))263# ... draw filled rectangles on overlay with alpha=60 ...264img_draw.paste(overlay, (0, 0), overlay)265```266 267The alpha value of 60 (out of 255) gives a ~24% opacity fill, keeping the underlying content readable.268 269### Colour assignment270 271Each unique label gets a random RGB colour, generated once per session:272 273```python274np.random.seed(42)275color_map[label] = (276    np.random.randint(50, 255),277    np.random.randint(50, 255),278    np.random.randint(50, 255)279)280```281 282The seed is fixed at 42, so label colours are deterministic across runs — `title` will always get the same colour, `text` always another. The lower bound of 50 prevents colours that are too dark to see against the fill.283 284Title regions get a thicker outline (width=5) than other regions (width=3) to give them visual prominence.285 286---287 288## Math Rendering Pipeline289 290Getting LaTeX from the model to display correctly in the browser involves three components working together.291 292### The markdown/math conflict293 294Standard markdown processors interpret `_` as italic and `*` as bold. Raw LaTeX like `$a_1 + a_2^*$` would be mangled before MathJax ever sees it.295 296The solution is `pymdownx.arithmatex` — a markdown extension that extracts math expressions **before** markdown processing, processes the surrounding text, then reinserts the math wrapped in MathJax-compatible delimiters:297 298```299Input:  Some text with $a_1 + a_2$ inline.300 301After arithmatex + markdown:302<p>Some text with <span class="arithmatex">\(a_1 + a_2\)</span> inline.</p>303```304 305The `_` inside the math is never touched by the markdown processor.306 307### Delimiter pre-conversion308 309The model outputs `\[...\]` for display math and `\(...\)` for inline math. But `pymdownx.arithmatex` only recognises `$...$` and `$$...$$` by default. Worse, if `\[...\]` is passed directly to the markdown processor, the backslashes are stripped first — before arithmatex can intercept them — leaving bare `[...]` brackets in the output.310 311`to_math_html()` therefore pre-converts the model's native delimiters before calling `markdown()`:312 313```python314text = re.sub(r'\\\[(.+?)\\\]', r'$$\1$$', text, flags=re.DOTALL)315text = re.sub(r'\\\((.+?)\\\)', r'$\1$', text)316```317 318After this step, arithmatex sees `$$...$$` and `$...$`, protects the content from markdown, and wraps it in `\[...\]` and `\(...\)` for MathJax to render.319 320### MathJax configuration321 322MathJax is loaded once in the page `<head>` and configured to process `\(...\)` for inline math and `\[...\]` for display math — matching the output format of arithmatex:323 324```javascript325window.MathJax = {326  tex: {327    inlineMath:  [['\\(', '\\)']],328    displayMath: [['\\[', '\\]']],329    processEscapes: true,330    tags: 'ams'331  }332};333```334 335`tags: 'ams'` enables automatic equation numbering for `align`, `equation`, and similar environments.336 337### Re-typesetting on update338 339MathJax processes the page once on load. When Gradio updates the HTML component with new content, MathJax needs to be told to process the new content:340 341```python342submit_event.then(343    fn=None,344    js="() => setTimeout(() => { if(window.MathJax) MathJax.typesetPromise(); }, 300)"345)346```347 348The 300ms delay gives Gradio time to finish updating the DOM before MathJax scans it.349 350---351 352## Known Quirks and Workarounds353 354### `\coloneqq` and `\eqqcolon`355 356These LaTeX commands (`≔` and `=:`) from the `mathtools` package appear frequently in academic papers but are not available in MathJax's default TeX configuration. Rather than loading the full `mathtools` package, they are substituted at the text level:357 358```python359text = text.replace('\\coloneqq', ':=').replace('\\eqqcolon', '=:')360```361 362If you need proper rendering of these symbols, add `require: { package: ['mathtools'] }` to the MathJax configuration.363 364### Flash Attention initialisation warning365 366On startup you will see:367 368```369You are attempting to use Flash Attention 2.0 with a model not initialized on GPU.370```371 372This is because the model loads onto CPU first (`from_pretrained`) then moves to GPU (`.cuda()`). Flash Attention 2 prefers direct GPU initialisation. The warning is harmless — inference works correctly. To silence it, add `device_map="cuda"` to the `from_pretrained` call.373 374### Model type mismatch warning375 376```377You are using a model of type deepseek_vl_v2 to instantiate a model of type DeepseekOCR2.378```379 380The model's config file on HuggingFace declares `model_type: deepseek_vl_v2` but the custom code registers a `DeepseekOCR2` class. Because `trust_remote_code=True` is set, the correct class is loaded regardless. The warning can be ignored.381 382### `eval()` on model output383 384Bounding box coordinates are parsed with Python's `eval()`:385 386```python387coords = eval(ref[2])388```389 390The model outputs coordinates as a Python list literal e.g. `[[45, 12, 820, 48]]`. This is safe in this context since the model runs locally, but worth noting if the architecture ever changes to process untrusted remote model output.391 392### Zone.Identifier files in examples/393 394Files copied from Windows to WSL2 may have accompanying `.Zone.Identifier` metadata files (e.g. `image.png:Zone.Identifier`). These are Windows security zone markers and are harmless — Gradio ignores them when loading examples.395 396---397 398## VRAM Usage and Quantized Models399 400### The 8GB problem401 402The full-precision BF16 model (`deepseek-ai/DeepSeek-OCR-2`) consumes approximately **7.9GB of VRAM** on load, leaving only ~250MB free on an 8GB GPU (e.g. RTX 3070). This headroom is insufficient for inference on complex documents:403 404- Each patch adds tokens to the KV cache and activations405- A 6-patch document can exhaust the remaining VRAM406- When VRAM is full, PyTorch spills to system RAM — which is 50–100× slower407- Symptom: GPU-Util drops to ~24%, power draw falls to ~47W (waiting on memory, not computing)408 409You can confirm this with `watch -n 1 nvidia-smi` during inference. Near-full VRAM with low GPU utilisation is the telltale sign.410 411### Quantized alternatives on HuggingFace412 413To switch models, change `MODEL_NAME` in `app.py`. Three options are available as of March 2026:414 415| Model | Format | VRAM | Notes |416|---|---|---|---|417| `deepseek-ai/DeepSeek-OCR-2` | BF16 (full) | ~8GB | Original, highest accuracy |418| `richarddavison/DeepSeek-OCR-2-FP8` | FP8 dynamic | ~3.5GB | ~50% reduction; requires Ampere GPU or newer (RTX 30xx qualifies); 3,750 downloads/mo |419| `mzbac/DeepSeek-OCR-2-8bit` | 8-bit | ~4GB | Same stack (torch 2.6, flash-attn 2.7.3, Python 3.12); explicitly supports dynamic resolution (0–6 patches); 140 downloads/mo |420 421**Not applicable to NVIDIA GPUs:**422- `mlx-community/DeepSeek-OCR-2-*` — Apple Silicon only (MLX framework)423 424**Not recommended:**425- `WHY2001/DeepSeek-OCR-4bit-Quantized` — 17 downloads/month, not well tested426 427### What does not exist (as of March 2026)428 429- GGUF of DeepSeek-OCR-2 (GGUF repos on HuggingFace are for v1 only)430- GPTQ of DeepSeek-OCR-2431- AWQ of DeepSeek-OCR-2432 433### Switching models434 435Change the single constant in `app.py` and restart:436 437```python438# FP8 — recommended first try for 8GB GPUs439MODEL_NAME = 'richarddavison/DeepSeek-OCR-2-FP8'440 441# 8-bit — alternative with same toolchain442MODEL_NAME = 'mzbac/DeepSeek-OCR-2-8bit'443```444 445The model will be downloaded from HuggingFace on first use and cached locally.446