rukiyeberna/qwen2.5-vl-3b-food-extract-lora

Model Card for rukiyeberna/qwen2.5-vl-3b-food-extract-lora

Model Details

Model Description

This repository contains the parameter-efficient LoRA adapter weights for Qwen2.5-VL-3B-Instruct, fine-tuned for multimodal structured attribute extraction from food photography. The model analyzes dish images and outputs strictly formatted JSON including food presence verification, title, ingredients, and accompanying beverages.

• —Developed by: Rukiye Berna Turan
• —Model type: Vision-Language Model (VLM) LoRA Adapter
• —Language(s) (NLP): English
• —License: Apache-2.0
• —Finetuned from model: Qwen/Qwen2.5-VL-3B-Instruct

Model Sources

• —Repository: https://github.com/rukiyeberna/vlm-food-extractor
• —Base Model Card: https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct

Uses

Direct Use

Zero-shot and structured visual extraction of culinary attributes from food images. Designed to produce strict JSON schemas for menu parsing, dietary tracking, recipe indexing, and food dataset labeling.

Out-of-Scope Use

• —Medical allergen confirmation or clinical dietary advice.
• —Non-food visual document parsing.
• —Precise weight, volume, or caloric estimation.

Bias, Risks, and Limitations

• —The model may miss occluded, mixed, or interior ingredients that are not visually apparent on the plate surface.
• —Exact token matching can vary compared to colloquial culinary names (e.g., predicting tomato salad instead of caprese salad).

How to Get Started with the Model

import torch
from PIL import Image
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

BASE_MODEL = "Qwen/Qwen2.5-VL-3B-Instruct"
ADAPTER_PATH = "rukiyeberna/qwen2.5-vl-3b-food-extract-lora"

# Load base model & LoRA adapter
base_model = AutoModelForImageTextToText.from_pretrained(
    BASE_MODEL,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, ADAPTER_PATH)
processor = AutoProcessor.from_pretrained(ADAPTER_PATH)
model.eval()

# Load image and build prompt
image = Image.open("sample_food.jpg").convert("RGB")
prompt = """Analyze this image and return only valid JSON with exactly these keys:
- is_food
- image_title
- food_items
- drink_items"""

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": prompt},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(model.device)

with torch.inference_mode():
    generated_ids = model.generate(
        **inputs,
        max_new_tokens=128,
        do_sample=False,
        repetition_penalty=1.15,
        no_repeat_ngram_size=3
    )

output = processor.batch_decode(
    generated_ids[:, inputs.input_ids.shape[1]:], 
    skip_special_tokens=True
)[0]

print(output)

Training Details

Training Data

Trained on mrdbourke/FoodExtract-1k-Vision (1,510 training samples) preprocessed into LLaVA-style conversational format (image + structured JSON response).

Training Hyperparameters

• —Regime: LoRA PEFT (Vision encoder and LLM backbone frozen)[cite: 1]
• —Precision: bfloat16[cite: 1]
• —LoRA Rank ($r$): 32[cite: 1]
• —LoRA Alpha ($\alpha$): 64
• —LoRA Dropout: 0.05[cite: 1]
• —Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj[cite: 1]
• —Learning Rate: $1 \times 10^{-4}$ (Cosine Decay with warmup)[cite: 1]
• —Global Steps: 48 steps (1 epoch)[cite: 1]
• —Loss Reduction: Initial 1.7121 $\rightarrow$ Final 0.8888 (48.08% convergence reduction)[cite: 1]

Evaluation

Testing Data & Metrics

Evaluated on unseen food test images across structured schema integrity, sequence similarity, and fuzzy set-based overlap[cite: 1]:

• —JSON Parsing Success Rate[cite: 1]
• —`is_food` Classification Accuracy[cite: 1]
• —Title Character Similarity (SequenceMatcher)[cite: 1]
• —Set-based Item $F_1$ Score (Precision / Recall)[cite: 1]
• —ROUGE-1 / ROUGE-L Overlap

Technical Specifications

Architecture

• —Base Architecture: Qwen2.5-VL (3B Parameters)[cite: 1]
• —Adapter Type: Low-Rank Adaptation (LoRA) on Self-Attention and MLP Projections[cite: 1]
• —Hardware: Single GPU (NVIDIA T4 / A100 environment)

Framework Versions

• —PEFT: 0.19.1
• —Transformers: 4.49.0+
• —PyTorch: 2.x
• —qwen-vl-utils: Latest