MayaKD/qwen2-vl-7b-nutrition-vllm

Qwen2-VL-7B — Nutrition Table Detection

Fine-tuned `Qwen/Qwen2-VL-7B-Instruct` that localizes nutrition tables in product photographs. Given an image, it emits bounding boxes as text using Qwen2-VL's native grounding tokens — no detection head, no anchor boxes.

These are merged full weights at the repo root, so vLLM and from_pretrained can load this repo directly. For training artifacts (LoRA adapters, per-stage checkpoints, optimizer state) see `MayaKD/qwen2-vl-7b-nutrition`.

Results

Evaluated on all 123 validation samples of openfoodfacts/nutrition-table-detection:

Why the threshold metrics are bounds. They were computed with a matching rule that counted every IoU-matrix cell above 0.5 rather than pairing each ground-truth box with at most one prediction, so duplicate detections on one table overcounted true positives. The corrected implementation does greedy one-to-one matching; the bias it removes is strictly upward, so true values sit at or below those shown. Mean IoU is threshold-free, never used that path, and is the metric to judge this model on.

Serving throughput on a single GPU, measured with real async concurrency:

Throughput peaks at c=8 and collapses at c=16 as the scheduler saturates.

Prompts

Send the strings the model was fine-tuned on, exactly as written below. How much accuracy is lost by deviating has not been measured cleanly, so treat these as part of the interface rather than as tunable inputs.

system: You are a Vision Language Model specialized in interpreting visual data from product images.
        Your task is to analyze the provided product images and detect the nutrition tables in a certain format.
        Focus on delivering accurate, succinct answers based on the visual information. Avoid additional explanation unless absolutely necessary.

user:   Detect the bounding boxes of all nutrition tables in the image.

Usage

import torch
from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
from qwen_vl_utils import process_vision_info

REPO = "MayaKD/qwen2-vl-7b-nutrition-vllm"
SYSTEM = (
    "You are a Vision Language Model specialized in interpreting visual data from product images.\n"
    "Your task is to analyze the provided product images and detect the nutrition tables in a certain format.\n"
    "Focus on delivering accurate, succinct answers based on the visual information. "
    "Avoid additional explanation unless absolutely necessary."
)
PROMPT = "Detect the bounding boxes of all nutrition tables in the image."

model = Qwen2VLForConditionalGeneration.from_pretrained(
    REPO, dtype=torch.bfloat16, device_map="auto",
)
processor = Qwen2VLProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": [
        {"type": "image", "image": "https://static.openfoodfacts.org/images/products/27563564/2.jpg"},
        {"type": "text", "text": PROMPT},
    ]},
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, _ = process_vision_info(messages)
inputs = processor(text=[text], images=image_inputs, return_tensors="pt").to(model.device)

out_ids = model.generate(**inputs, max_new_tokens=256, do_sample=False)
trimmed = [o[len(i):] for i, o in zip(inputs.input_ids, out_ids)]
print(processor.batch_decode(trimmed, skip_special_tokens=False)[0])

vLLM

python -m vllm.entrypoints.openai.api_server \
  --model MayaKD/qwen2-vl-7b-nutrition-vllm \
  --dtype bfloat16 --host 0.0.0.0 --port 8000

Output format

Each detection is emitted as:

<|object_ref_start|>nutrition-table<|object_ref_end|><|box_start|>(x0, y0),(x1, y1)<|box_end|>

Coordinates are (x_min, y_min), (x_max, y_max) on a 0–1000 scale, relative to image dimensions. Multiple detections concatenate. Images with no table produce "No table found."

Training

LoRA (r=16, α=32) applied to the frozen base in two stages. Adapters are merged into the base between stages, so each stage starts from clean full weights rather than stacked adapters.

Trained with SFTTrainer + accelerate on 2× RTX Pro 6000, BF16 with Flash Attention 2 and fused AdamW. Best checkpoint by eval_loss, early stopping patience 3. Image resolution min_pixels=784, max_pixels=705600; sequence length is deliberately not capped, since truncation corrupts image tokens.

A three-stage variant reached 0.893 mean IoU in exploratory runs, indicating headroom above this checkpoint. Those weights were not preserved; this two-stage model is the reproducible artifact.

Limitations

• —Annotation convention. The training data labels only the official EU-format nutrition declaration (the standardized "Per 100 g" table). Colorful summary panels and "per portion" columns are not annotated even when visible, and the model reproduces this — it will ignore non-standard nutrition displays by design.
• —Out of distribution on crowded scenes. Multi-product shelf images with many labels in frame degrade noticeably. Single-product images are the intended input.
• —Prompt sensitivity is uncharacterized. Match the training strings above, but note that the sensitivity has not been measured cleanly in either direction — see Prompts.
• —English-language packaging dominates the training distribution.
• —Evaluation is on 123 validation samples, which is small — treat the metrics as indicative rather than tight estimates.

Links

• —Code, benchmarks, and full writeup: github.com/MKDehdashti/qwen2-vl-nutrition-table
• —Training artifacts: `MayaKD/qwen2-vl-7b-nutrition`
• —Dataset: `openfoodfacts/nutrition-table-detection`

License

Apache-2.0, inherited from Qwen2-VL-7B-Instruct. Project code is MIT.