Modelpublic

Remidesbois/surya-ocr-2-poneglyph-bbox

sourceHugging Faceopenrailupdated 15d agoView on Hugging Face

0likes22downloads

Model Card

surya-ocr-2-poneglyph-bbox

Surya OCR 2 fine-tuned for One Piece manga bubble text plus bounding boxes

This model reads a full manga page and emits one line per dialogue bubble:

text

Text content [x1,y1,x2,y2]

Coordinates are normalized to [0, 1000] on the resized page image.

</div>

Why Surya For BBox

The upstream Surya OCR 2 card documents bbox-capable outputs in three relevant paths:

—OCR output includes per-block polygon, axis-aligned bbox, confidence, and reading order.
—surya_detect returns text-line bboxes and polygons.
—surya_layout returns layout boxes, labels, reading order, and bbox values.

This fine-tune uses the Hugging Face image-text-to-text Surya OCR 2 model and teaches the generated text stream to match the existing Poneglyph bbox contract.

Benchmark: Surya vs LightOn BBox Poneglyph

Metric	Surya OCR 2 fine-tuned	LightOn bbox Poneglyph	Winner
CER	1.49%	0.42%	LightOn
WER	3.44%	1.73%	LightOn
Mean IoU	92.36%	72.98%	Surya
Median IoU	94.11%	73.43%	Surya
F1 @ IoU=0.5	98.36%	77.74%	Surya
Precision @ 0.5	98.58%	78.00%	Surya
Recall @ 0.5	98.38%	77.54%	Surya
Detection Rate	98.38%	77.54%	Surya
Combined Score	0.972	0.855	Surya
Avg Inference	2.75s/page	5.61s/page	Surya

Surya Fine-Tuned Snapshot

Metric	Score
CER	1.49%
WER	3.44%
Mean IoU	92.36%
Median IoU	94.11%
F1 @ IoU=0.3	98.41%
F1 @ IoU=0.5	98.36%
F1 @ IoU=0.75	96.67%
Detection Rate	98.38%
Combined Score	0.972
Avg Inference	2.75s/page

Combined score:

text

0.4 * (1 - CER) + 0.3 * F1@0.5 + 0.2 * MeanIoU + 0.1 * DetectionRate

Dataset

Source data comes from the Poneglyph Supabase bulles table, filtered to validated annotations, grouped at page level, and split by id_page to prevent page leakage.

Split	Pages	Bubbles
train	772	6935
val	166	1431
test	166	1501

Preprocessing:

—Full page image resized to 1540px longest side.
—JPEG quality 95.
—Bubble boxes normalized to [0, 1000].
—Target order follows the stored manga reading order.
—Target text uses one strict line per bubble.

How To Use

bash

pip install torch pillow transformers accelerate

python

import re
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

MODEL_ID = "Remidesbois/surya-ocr-2-poneglyph-bbox"
PROMPT = "Tu es un moteur OCR de page de manga. Extrais les textes visibles associés aux zones annotées et leur bbox normalisée.\n\nFormat de sortie STRICT, une zone par ligne :\nTexte exact [x1,y1,x2,y2]\n\nRègles :\n- Coordonnées entières normalisées entre 0 et 1000 dans le repère de l'image.\n- Respecte l'ordre de lecture japonais : haut droite vers bas gauche ; dans une case, droite vers gauche.\n- Garde le français et ne traduis pas.\n- Rétablis une casse naturelle et corrige uniquement les erreurs OCR évidentes de ponctuation/casse.\n- Ignore les zones vides ou illisibles.\n- N'ajoute aucun JSON, Markdown, commentaire, préfixe ni suffixe.\n- Chaque ligne doit se terminer exactement par une bbox au format [x1,y1,x2,y2]."

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

image = Image.open("page.jpg").convert("RGB")
image.thumbnail((1540, 1540), Image.Resampling.LANCZOS)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "page.jpg"},
            {"type": "text", "text": PROMPT},
        ],
    }
]

prompt = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)
inputs = processor(text=[prompt], images=[image], return_tensors="pt")
inputs = {
    k: v.to(model.device, dtype=torch.bfloat16) if v.is_floating_point() else v.to(model.device)
    for k, v in inputs.items()
}

with torch.inference_mode():
    output_ids = model.generate(**inputs, max_new_tokens=2048, do_sample=False)

generated = output_ids[0, inputs["input_ids"].shape[1]:]
text = processor.decode(generated, skip_special_tokens=True).strip()
print(text)

pattern = re.compile(r"(.+?)\s*\[(\d+),(\d+),(\d+),(\d+)\]")
bubbles = [
    {"text": m.group(1).strip(), "bbox": [int(m.group(i)) for i in range(2, 6)]}
    for line in text.splitlines()
    if (m := pattern.match(line.strip()))
]

Training

The training package used for this model lives in:

text

docker_scripts/finetune_surya_ocr_bbox

Pipeline:

bash

python run_pipeline.py --dry-run --check-remote
python run_pipeline.py

The run exports the dataset, fine-tunes Surya OCR 2 with LoRA/DoRA, benchmarks the held-out test split, benchmarks Remidesbois/LightonOCR-2-1b-poneglyph-bbox on the same pages, writes this README, and uploads the final merged model when HF_TOKEN is available.

Limitations

—Domain-specific: trained for One Piece manga pages.
—Text language: French annotations.
—Output is a generated text contract, so malformed lines are possible and should be parsed defensively.
—The model returns normalized bbox coordinates, not pixel coordinates.
—The LightOn comparison is only valid when both models are evaluated on the same exported test split.

Base Model

Fine-tuned from `datalab-to/surya-ocr-2`. The base model uses Surya OCR 2 / Qwen3.5 image-text-to-text architecture.

Fine-tuned by [Remidesbois](https://huggingface.co/Remidesbois).