msaid1976/SmolVLM-Instruct-Navigation-FineTuned

SmolVLM Navigation Assistant 🦯

<div align="center">

  

Fine-tuned vision-language model for blind navigation assistance

Quick Start • Performance • Usage • Training • Citation

</div>

📋 Overview

Fine-tuned SmolVLM-500M-Instruct for vision-based navigation assistance for blind and visually impaired users. Developed as a Master's thesis project at Asia Pacific University.

Key Results:

• —🎯 91.6% BERTScore (semantic accuracy)
• —🚀 +3483% BLEU-1 improvement over baseline
• —⚡ 0.5-1s inference time
• —💾 2-4GB VRAM requirement
• —📊 p < 0.001 statistical significance

Author: Mohammad Mohamed Said Aly Amin Supervisor: Dr. Raheem Mafas Institution: Asia Pacific University Program: Master's in Data Science & Business Analytics

✨ Features

Three Navigation Modes

Technical Highlights

• —✅ Real-time inference on consumer GPUs
• —✅ Low memory footprint (2-4GB VRAM)
• —✅ Statistically validated improvements
• —✅ Production-ready deployment
• —✅ QLoRA efficient fine-tuning (1.84% parameters)

📊 Performance

Evaluation Results (500 samples)

Statistical Validation: All improvements significant at p < 0.001 (paired t-test, n=500)

Loss Convergence

• —Initial Training Loss: 0.29 → Final: 0.12 (58% reduction)
• —Initial Val Loss: 0.24 → Final: 0.13 (46% reduction)

🚀 Quick Start

Installation

pip install transformers torch pillow accelerate

Basic Usage

from transformers import Idefics3ForConditionalGeneration, AutoProcessor
from PIL import Image
import torch

# Load model
model = Idefics3ForConditionalGeneration.from_pretrained(
    "msaid1976/SmolVLM-Instruct-Navigation-FineTuned",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(
    "msaid1976/SmolVLM-Instruct-Navigation-FineTuned",
    trust_remote_code=True
)

# Prepare input
image = Image.open("scene.jpg")
messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "What do you see?"}
    ]
}]

# Generate
prompt = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = processor(text=prompt, images=[image], return_tensors="pt")
inputs = {k: v.to("cuda") for k, v in inputs.items()}

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=150,
        do_sample=False,
        pad_token_id=processor.tokenizer.eos_token_id,
        eos_token_id=processor.tokenizer.eos_token_id
    )

response = processor.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)
print(response)

💡 Usage Examples

FOCUSED: Spatial Queries

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Is there a chair to the left of the table?"}
    ]
}]
# Output: "Yes, there is a chair to the left of the table."

SCENE: Environment Description

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "Describe the scene in front of me."}
    ]
}]
# Output: "The scene shows a living room with a brown sofa on the left, 
# a wooden coffee table in the center, and a TV on the wall..."

OCR: Text Reading

messages = [{
    "role": "user",
    "content": [
        {"type": "image"},
        {"type": "text", "text": "What text is on the sign?"}
    ]
}]
# Output: "The sign says 'EXIT' in red letters."

Memory Optimization

# 8-bit quantization (reduces to ~2GB VRAM)
model = Idefics3ForConditionalGeneration.from_pretrained(
    "msaid1976/SmolVLM-Instruct-Navigation-FineTuned",
    load_in_8bit=True,
    device_map="auto"
)

# Batch processing
inputs = processor(
    text=[prompt1, prompt2, prompt3],
    images=[[img1], [img2], [img3]],
    return_tensors="pt",
    padding=True
)

🛠️ Training Details

Configuration

Dataset

Size: 10,000+ samples across three modes

Sources:

• —GQA Enhanced (spatial reasoning)
• —Localized Narratives (scene descriptions)
• —Visual Genome (object relationships)
• —TextCaps (text-in-image)
• —VizWiz (accessibility focus)

💻 Hardware Requirements

Recommended for Inference: RTX 3060+ or equivalent

⚠️ Limitations

1. 1.Scope: Optimized for navigation; may underperform on general VQA
2. 2.Image Quality: Best with well-lit, clear images
3. 3.OCR: Works best with printed text; struggles with handwriting
4. 4.Speed: Requires GPU for real-time use (CPU: 10-20s/image)
5. 5.Language: English only

Safety Notice

⚠️ This is an assistive tool, not a replacement for traditional navigation aids. Users should:

• —Combine with cane, guide dog, or other mobility aids
• —Exercise human judgment
• —Test in safe environments first
• —Be aware of potential errors

🎓 Model Card

Model Details

• —Type: Vision-Language Model (Idefics3)
• —Parameters: 500M total, 42M trainable (1.84%)
• —Input: Image + Text
• —Output: Text
• —License: Apache 2.0

Intended Use

Primary:

• —Navigation assistance for blind/visually impaired
• —Spatial reasoning and object localization
• —Scene understanding and description
• —Text recognition in natural environments
• —Accessibility research

Out of Scope:

• —Medical diagnosis
• —Autonomous navigation without human oversight
• —Real-time video processing
• —General-purpose VQA (use base model)

Ethical Considerations

• —Designed to enhance independence, not replace human judgment
• —May have biases from English-only training data
• —Requires validation in real-world scenarios
• —Processes images locally (no data collection)

📖 Citation

@misc{alqahtani2025smolvlm_navigation,
  author = {Alqahtani, Muhammad Said},
  title = {SmolVLM Navigation Assistant: Fine-tuned for Blind Navigation},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/msaid1976/SmolVLM-Instruct-Navigation-FineTuned}}
}

@mastersthesis{alqahtani2025thesis,
  author = {Alqahtani, Muhammad Said},
  title = {An Efficient Multi-Object Detection and Smart Navigation Using Vision Language Models for Visually Impaired},
  school = {Asia Pacific University of Technology and Innovation},
  year = {2025},
  address = {Kuala Lumpur, Malaysia}
}

🙏 Acknowledgments

Supervision:

• —Dr. Raheem Mafas (Research Supervisor)
• —Asia Pacific University

Technical:

• —HuggingFace Team (base model & libraries)
• —Unsloth (training framework)
• —NVIDIA (GPU hardware)

Datasets:

• —Stanford Visual Genome
• —GQA, VizWiz, TextCaps
• —Localized Narratives

📫 Contact

Author: Mohammad Mohamed Said Aly Amin Institution: Asia Pacific University Issues: Model Discussions

<div align="center">

Made with ❤️ for accessibility and inclusion

 

Empowering independence through AI-powered vision assistance

</div>