CoolFace
Apppublic

kalpkanungo/SceneGraphNet

sourceHugging Faceupdated 6mo agoView on Hugging Face
0likes
App README

🧠 Scene Graph Generator (Multimodal AI System)

A multimodal computer vision system that takes an input image, detects objects, predicts relationships between them, constructs a structured scene graph, and generates a natural language description of the scene.

🔗 Live Demo (Hugging Face Spaces): https://huggingface.co/spaces/<your-username>/scene-graph-generator

🚀 Features

  • —🖼️ Object Detection using DETR (ResNet-50)
  • —🔗 Relationship Prediction (Custom Trained Model)
  • —📐 Spatial Reasoning (Hybrid AI with Geometry Rules)
  • —🧩 Scene Graph Construction (Directed Graph)
  • —📊 Graph Visualization (NetworkX + Matplotlib)
  • —🧠 Graph-to-Text Generation (FLAN-T5)
  • —🌐 Interactive UI (Gradio)
  • —☁️ Deployed on Hugging Face Spaces (CPU)

🧠 How It Works (End-to-End Pipeline)

1. Input

  • —User uploads an image (JPG/PNG) via Gradio UI
  • —Image is converted from PIL → OpenCV format

2. Object Detection

  • —Uses facebook/detr-resnet-50 from Hugging Face
  • —Outputs:
  • —Object labels (COCO classes)
  • —Bounding boxes
  • —Confidence scores
  • —Applies threshold (≥ 0.7) to filter noise

3. Pairwise Object Processing

  • —Generates object pairs using itertools.combinations
  • —Extracts bounding boxes for each pair
  • —Creates union region for relation inference
  • —Filters duplicate object pairs

4. Relationship Prediction

  • —Custom-trained classifier on Visual Genome subset (~10K samples)
  • —Predicts semantic relations:
  • —on, holding, behind, etc.
  • —Trained using PyTorch (10 epochs)

5. Spatial Reasoning (Hybrid AI)

  • —Uses bounding box geometry to compute:
  • —left_of, right_of, above, below, near
  • —Hybrid logic:
  • —Semantic relations from model (if confident)
  • —Otherwise fallback to spatial rules
  • —Reduces bias (e.g., “everything = on”)

6. Graph Construction

  • —Builds a directed graph (NetworkX DiGraph)
  • —Nodes → objects
  • —Edges → relationships
  • —Removes duplicates and limits edges for clarity

7. Graph Visualization

  • —Uses NetworkX + Matplotlib
  • —Displays:
  • —Directed edges with labels
  • —Clean layout for readability

8. Graph → Text (NLP)

  • —Uses google/flan-t5-small
  • —Converts structured triples into natural language

Example: laptop → on → table mouse → next_to → laptop

Output: "A laptop is placed on a table with a mouse next to it."

9. UI (Gradio)

  • —Upload image
  • —View:
  • —Scene graph
  • —Generated description
  • —Fully interactive and browser-based

🏗️ Tech Stack


- **Computer Vision:** DETR (Hugging Face Transformers)
- **Deep Learning:** PyTorch
- **Graph Processing:** NetworkX
- **NLP:** FLAN-T5
- **Image Processing:** OpenCV
- **Frontend/UI:** Gradio
- **Deployment:** Hugging Face Spaces

📁 Project Structure

scene-graph-generator/ │ ├── app.py ├── requirements.txt ├── README.md │ ├── src/ │ ├── pipeline.py │ ├── detection.py │ ├── spatialrules.py │ ├── relationshipinfer.py │ ├── scenegraph.py │ ├── visualization.py │ ├── textgeneration.py

⚙️ Installation (Local Setup)

bash
git clone https://github.com/<your-username>/scene-graph-generator.git
cd scene-graph-generator

pip install -r requirements.txt
python app.py