camino-gr/florence2-sam2-demo

Florence-2 + SAM Image Recognition Demo

🚀 Live Demo: https://huggingface.co/spaces/camino-gr/florence2-sam-demo

A combined pipeline that uses Florence-2 for image captioning and object detection, then feeds detected objects into SAM for high-quality segmentation masks. Includes a Gradio web UI for interactive use.

Architecture

Input Image
    │
    ▼
┌──────────────┐
│  Florence-2   │  → Caption + Object Detection (bounding boxes)
│  (Microsoft)  │
└──────┬───────┘
       │ bounding boxes
       ▼
┌──────────────┐
│    SAM      │  → Segmentation masks for each detected object
│    (Meta)     │
└──────┬───────┘
       │
       ▼
┌──────────────┐
│ Visualization │  → Overlay bboxes, masks, and captions on image
└──────┬───────┘
       │
       ▼
  Gradio Web UI

Features

• —Image Captioning — Generate natural language descriptions of images
• —Object Detection — Detect objects with bounding boxes using Florence-2
• —Instance Segmentation — Pixel-level masks via SAM for each detected object
• —Interactive Web UI — Upload images and see results instantly via Gradio

Setup

Prerequisites

• —Python 3.10+
• —~4GB disk space for model weights (downloaded on first run)

Installation

git clone https://github.com/caminogr/florence2-sam-demo.git
cd florence2-sam-demo
python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

Run the Demo

python app.py

Open http://localhost:7860 in your browser.

Usage

1. 1.Open the Gradio web UI
2. 2.Upload an image (or select an example)
3. 3.View results:
4. 4.Caption — natural language description
5. 5.Detection — image with bounding boxes and labels
6. 6.Segmentation — image with colored masks overlaid

Configuration

The pipeline runs on CPU by default. To use GPU, ensure CUDA is available — the code auto-detects and uses it.

Project Structure

florence2-sam-demo/
├── README.md
├── requirements.txt
├── pyproject.toml
├── src/
│   ├── __init__.py
│   ├── florence2_model.py    # Florence-2 inference wrapper
│   ├── sam_model.py         # SAM inference wrapper
│   ├── pipeline.py           # Combined pipeline
│   └── visualization.py      # Drawing bboxes, masks, captions
├── app.py                    # Gradio demo
├── examples/                 # Sample images
├── .gitignore
└── LICENSE

Screenshots

<!-- Add screenshots here --> Screenshots will be added after first successful run.

Models Used

License

MIT — see LICENSE.