Modelpublic

oddadmix/Nawah-VL-50M-grounding-ar-2ep

sourceHugging Faceapache-2.0updated 1mo agoView on Hugging Face

0likes25downloads

Model Card

Nawah-VL-50M — Arabic visual grounding (2 epochs)

A 50M-parameter vision-language model that takes an image and an Arabic label and returns a bounding box. Trained on `oddadmix/blip3-grounding-1m-arabic`.

This is the 2-epoch model. The earlier 3,000-step version is kept separately at `oddadmix/Nawah-VL-50M-grounding-ar`; both are published because they were measured independently.

Read the headline against the image-blind floor

`acc@0.5` is 0.654 — but a constant box that never looks at the image scores 0.262 on these same rows. 26% of the reference boxes cover more than half the image, so a quarter of any headline number here is free. The honest margin is ~0.39, not 0.65.

Metric	This model (2ep)	v1 (3k steps)	Image-blind floor
`acc@0.5`	0.654	0.498	0.262
`acc@0.25`	0.778	0.6805	—
`acc@0.75`	0.462	0.327	—
`mean_iou`	0.6034	0.4984	0.326
`format_ok`	0.9835	0.994	—
multibox P / R / F1	0.6405 / 0.6101 / 0.6249	0.4747 / 0.4488 / 0.4614	—

n = 2000, held-out shard. The image-blind floor is a constant 00 00 99 99 box; a constant box grid-searched on the test rows themselves only reaches 0.297. The un-finetuned base scores 0.0 because it emits no parseable box at all — that is a format floor, not a spatial one, which is why it is the wrong number to brag against.

Accuracy is mostly a function of object size

From the 200 dumped predictions (identical rows for both models, so this is like-for-like):

Gold box area	n	v1 `acc@0.5`	2ep `acc@0.5`	v1 mIoU	2ep mIoU
tiny, <5% of image	50	0.100	0.280	0.197	0.271
small, 5–20%	40	0.350	0.600	0.411	0.514
medium, 20–50%	36	0.750	0.917	0.638	0.788
large, >50%	57	0.965	0.982	0.908	0.944
multi-box rows	17	0.000	0.176	0.045	0.223

Two things worth knowing before you use this:

—Small objects are where it fails. Large objects are essentially solved (0.982); tiny ones are not (0.280). In v1 the two smallest bands sat below the image-blind floor — it was doing scene-level attention, not localisation. They now clear it, but the gap to large objects is still the dominant error.
—Multi-box queries are weak. A label matching several objects should produce boxes joined by ; . v1 never did this (0.000); this model manages 0.176. Still the worst band.

format_ok 0.9835 means the output syntax is essentially solved — the remaining error is spatial, not syntactic.

Usage

python

from transformers import AutoModelForImageTextToText, AutoProcessor
from PIL import Image
import torch

repo = "oddadmix/Nawah-VL-50M-grounding-ar-2ep"
model = AutoModelForImageTextToText.from_pretrained(repo, dtype=torch.bfloat16).cuda().eval()
processor = AutoProcessor.from_pretrained(repo)

image = Image.open("photo.jpg").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": "قطة\nحدد الإطار المحيط."},
]}]
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=[text], images=[[image]], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
# -> "28 01 72 99"   (x1 y1 x2 y2 in 0-99 bins; multiple boxes joined by " ; ")

Output format. Coordinates are integers in 0–99 bins, x1 y1 x2 y2, relative to the image. Divide by 99 to get [0,1]. Bins rather than floats because they tokenise to exactly 4 tokens where floats cost 16 — which matters a great deal at 50M parameters. Round-trip error is at most half a bin.

Feed it original image bytes. The model is sensitive to re-encoding: re-saving one 158×150 test frame as JPEG q72 moved a correct prediction (00 00 99 16) to the whole image (00 00 99 99).

Training

Initialised from the v1 grounding checkpoint, itself initialised from the VQA-v3 checkpoint, so the model could already read Arabic against an image before it saw a box.


Data	247 shards, 1,976,371 rows (1 shard held out for eval)
Schedule	2 epochs = 15,440 updates, effective batch 256, cosine + 3% warmup
Stage	2 — projector + language model. The vision tower was frozen.
LR	projector 2e-4, LM 1e-4
Best checkpoint	step 15,000, eval_loss 1.8270 (v1's best was 2.1950)
Wall clock	252 min on one A6000 at ~134 samples/s

Targets are all boxes for a label at confidence ≥ 0.40, Arabic labels only, capped at 6 boxes; images capped at 512 px on the long side.

A note on the v1 → 2ep jump. v1's loss went flat over its last 500 steps and that was read as convergence. It was not — it was the cosine schedule decaying to zero. Given a fresh schedule the same model fell straight through that apparent floor, from 2.1950 to 1.8270, and acc@0.5 went 0.498 → 0.654 with it. Loss and IoU had come apart in v1, so the geometry eval is what settles these questions, not the loss curve.

What to try next

The vision tower has never been unfrozen. Given format_ok 0.9835 against tiny-object acc@0.5 0.280, the residual error is spatial, and a frozen tower is the obvious cap on spatial precision.