cmboulanger/nuextract3-toc

NuExtract3 — table-of-contents parser (LoRA fine-tune)

NuExtract3 (a Qwen3.5-based vision-language model), LoRA fine-tuned to extract structured table-of-contents entries from scanned TOC page images:

{
  "entries": [
    {
      "title": "Vorwort",
      "authors": [],
      "printed_page_number": "vii",
      "skip": true
    },
    {
      "title": "1. Introduction",
      "authors": ["Jane Doe"],
      "printed_page_number": "1",
      "skip": false
    }
  ]
}

skip marks a line that isn't an actual chapter — front matter (e.g. a preface, list of contributors), back matter (e.g. bibliography, index), or a part/section divider. It's still emitted as its own entry with its own title/page, never omitted; only its own true/false correctness is excluded from evaluation's precision/recall/F1 match key (see "Results" below) — the entry itself is still scored on title+page like any other.

The adapter has been merged into the base weights, so this checkpoint is a drop-in replacement for numind/NuExtract3 — same architecture, same AutoModelForImageTextToText/AutoProcessor loading path, same template-mode prompt contract (see below). No PEFT dependency needed to run it.

Trained and evaluated with the pipeline in finetune-nuextract-toc — see that repo's README.md and RESULTS.md for the full training/eval setup, dataset stats, and per-run history. The summary below mirrors its latest entry.

Intended use

Extracting {title, authors, printed_page_number, skip} entries from a scanned table-of-contents page (1-3 pages, rendered to an image no larger than 1200px on its longer side). Trained on German-language academic book TOCs sourced from the Deutsche Nationalbibliothek (DNB); expect degraded quality outside that domain (other languages, other document types, non-TOC pages).

How to use

Requires the exact same prompt template and instructions used during fine-tuning — copy common.py and prepare_data.py's render_pages_to_pngs from the training repo rather than reconstructing the prompt by hand, since even small prompt drift will hurt output quality. Minimal usage:

import json
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo_id = "cmboulanger/nuextract3-toc"
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo_id, trust_remote_code=True, dtype=torch.bfloat16
).to("cuda")

template = {"entries": [{"title": "verbatim-string", "authors": ["string"],
                          "printed_page_number": "verbatim-string", "skip": "boolean"}]}
instructions = "..."  # see common.py's _INSTRUCTIONS for the full text
messages = [{"role": "user", "content": [
    {"type": "text", "text": "Extract every table-of-contents entry from this page."},
    {"type": "image", "image": Image.open("toc-page.png").convert("RGB")},
]}]

inputs = processor.apply_chat_template(
    messages, template=json.dumps(template), instructions=instructions,
    enable_thinking=False, add_generation_prompt=True,
    tokenize=True, return_dict=True, return_tensors="pt",
).to("cuda")
generated = model.generate(**inputs, max_new_tokens=4096, do_sample=False)
completion = generated[:, inputs["input_ids"].shape[1]:]
print(processor.batch_decode(completion, skip_special_tokens=True)[0])

Results

Entry-level micro precision/recall/F1 on a 44-book held-out split (a prediction matches when normalized title and printed_page_number both match a ground-truth entry):

Fine-tuning's gain is almost entirely a precision story: the base model over-generates (1300 predicted vs. 1050 expected entries); fine-tuning brought predicted count in line with expected without giving up recall. Full dataset stats and per-book notes are in RESULTS.md in the training repo.

Training data

624 train / 44 eval books (15,372 / 1,050 TOC entries) from dnb-toc-ground-truth's pilot corpus of DNB table-of-contents scans, CC0-1.0 licensed. See the training repo's dataset_stats.py/RESULTS.md for the full breakdown (entries/book, page-count distribution, skip ratio).