small-models-for-glam/bl-crop-tighten-rfdetrseg-clip10

bl-crop-tighten-rfdetrseg-clip10

An instance-segmentation model that tightens the loose ABBYY picture-block crops in the British Library Book Images collection. A medium crop typically carries lines of body text above the art and a printed caption below; this model finds the actual illustration(s) inside the crop and returns boxes and masks. Masks matter for the non-rectangular cases — circular vignettes, medallions, irregular ornaments — where a tight box still carries page background.

It is a crop tightener, not a page-level illustration detector: inputs are crops that are already known to contain art. Fine-tuned from Roboflow/rf-detr-seg-medium.

Training

Distilled from tiiuae/Falcon-Perception-0.6B weak labels on bl-crop-tighten-v1: the teacher ran open-vocabulary segmentation over 8,400 stratified crops (embellishments, plates, medium, plus plain covers as negatives), a filter pass kept 7,565, and the student trained on those with book-safe splits (no book appears in both train and test). No human labels were used in training.

Evaluation — human-checked, not metric-only

On a 40-image random sample judged blind, 97.4% of the student's predictions were acceptable — statistically matching its teacher (95.0% on the same images) at roughly 12x the throughput (~70 img/s on a single L4). Running the full 1,019,266-crop corpus cost $3.24.

The model is domain-specific. On an out-of-domain probe (Encyclopaedia Britannica plates, which hold 10-20 figures each where BL crops average close to one) acceptability fell to 55% — the failure mode is dense multi-figure pages, inherited from the teacher, which shares the same blind spot. Use it on this collection or visually similar single-figure book crops; do not expect it to generalise to crowded plates.

Usage

Requires transformers, torch, torchvision:

import torch
from PIL import Image
from transformers import AutoImageProcessor, RfDetrForInstanceSegmentation

repo = "small-models-for-glam/bl-crop-tighten-rfdetrseg-clip10"
model = RfDetrForInstanceSegmentation.from_pretrained(repo)
processor = AutoImageProcessor.from_pretrained(repo)

image = Image.open("crop.jpg").convert("RGB")
inputs = processor(images=image, return_tensors="pt")
with torch.no_grad():
    outputs = model(**inputs)
result = processor.post_process_instance_segmentation(
    outputs, target_sizes=[image.size[::-1]], threshold=0.3
)[0]
# result["segmentation"]: HxW instance map; result["segments_info"]: per-instance scores

Predictions at low thresholds keep many low-confidence instances; filter at score >= 0.3 for display use. For the covered corpus you do not need to run inference at all — the full run is published as the `crop_masks` config of the source dataset.

Built by Daniel van Strien; source images digitised by the British Library.