hungvtm/qwen3vl-food-lora
qwen3vl-food-lora
A LoRA adapter fine-tuning Qwen3-VL-4B-Instruct (4-bit, via Unsloth) to analyze a food photo and return a structured description: dish name, food type, ingredients, cooking method, portion size, and nutrition (calories, protein, fat, carbohydrates).
Trained on Codatta/MM-Food-100K, 100K real user-submitted food photos spanning homemade, restaurant, raw, and packaged foods.
Full write-up: Building a Food Nutrition Estimation AI
Prompt format
The model was trained on a single fixed instruction, so use it verbatim for best results:
Analyze this food image and provide a complete description including:
dish name, food type, main ingredients, cooking method, portion sizes,
and nutritional information (calories, protein, fat, carbohydrates).Example output:
**Dish Name:** Vegetable Rice
**Food Type:** Homemade food
**Ingredients:** rice, green onions, mushrooms, potatoes
**Cooking Method:** boiled and mixed
**Portion Size:** rice:300g, vegetables:100g
**Nutritional Information:** Calories: 350.0 kcal, Protein: 10.0 g, Fat: 5.0 g, Carbohydrates: 60.0 gUsage (Unsloth — matches training)
from unsloth import FastVisionModel
from transformers import TextStreamer
from PIL import Image
model, tokenizer = FastVisionModel.from_pretrained(
model_name="unsloth/Qwen3-VL-4B-Instruct-bnb-4bit",
load_in_4bit=True,
)
model.load_adapter("hungvtm/qwen3vl-food-lora")
FastVisionModel.for_inference(model)
FOOD_PROMPT = (
"Analyze this food image and provide a complete description including: "
"dish name, food type, main ingredients, cooking method, portion sizes, "
"and nutritional information (calories, protein, fat, carbohydrates)."
)
image = Image.open("food.jpg").convert("RGB")
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": FOOD_PROMPT}]}]
input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
inputs = tokenizer(image, input_text, add_special_tokens=False, return_tensors="pt").to("cuda")
streamer = TextStreamer(tokenizer, skip_prompt=True)
_ = model.generate(**inputs, streamer=streamer, max_new_tokens=512, temperature=0.3, min_p=0.1)Usage (plain 🤗 transformers + peft)
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel
from PIL import Image
import torch
base_id = "unsloth/Qwen3-VL-4B-Instruct-bnb-4bit"
model = Qwen3VLForConditionalGeneration.from_pretrained(base_id, torch_dtype=torch.float16, device_map="cuda")
model = PeftModel.from_pretrained(model, "hungvtm/qwen3vl-food-lora")
processor = AutoProcessor.from_pretrained("hungvtm/qwen3vl-food-lora")
image = Image.open("food.jpg").convert("RGB")
prompt = (
"Analyze this food image and provide a complete description including: "
"dish name, food type, main ingredients, cooking method, portion sizes, "
"and nutritional information (calories, protein, fat, carbohydrates)."
)
messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": prompt}]}]
text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=text, return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=512, temperature=0.3)
print(processor.decode(out[0], skip_special_tokens=True))Training details
Loss curves
Training loss dropped from ~2.3 to under 0.1 within the first ~200 steps and stayed flat over ~1,200 steps; validation loss decreased from 0.119 to 0.110 across the run, indicating the model wasn't just memorizing the training set.
Evaluation
Exact-match dish-name accuracy on 20 held-out validation samples: 50%. However, manually reviewing the "misses" shows the metric understates real quality — 30% of predictions are the same dish under different wording (e.g. "coconut milk" → "coconut milk drink", "rice bowl with egg and meat" → "rice with meat and egg"), and only 20% name a genuinely different dish (e.g. "cold noodle salad" → "noodle soup").
Ingredient lists, cooking method, and portion-size estimates are consistently close to ground truth on packaged and homemade foods with a visible reference point; exact macro values (calories/protein/fat/carbs) are closer to plausible ranges than exact matches, since identical dishes can vary by hundreds of calories depending on how much oil went into cooking them.
Known limitations
- Exact macro estimation (calories, protein, fat, carbs) depends on hidden factors — oil absorbed during cooking, exact portion weight, recipe variation — that aren't recoverable from a single image. Values are plausible-range estimates, not lab measurements.
- Portion size for unpackaged food is visually noisy; a monocular depth map (see the training notebook's optional Depth Anything V2 section) helps for volumetric foods but can overestimate small, flat items like crackers or scattered nuts.
- Trained on a 5,000-sample subset of MM-Food-100K, not the full 100K.
Framework versions
- PEFT 0.18.1 · Unsloth · Transformers 5.0.0 · PyTorch 2.10.0+cu128 · Datasets 4.8.3