CoolFace
Apppublic

thesparshsaxena/Visual-Recall

sourceHugging Faceupdated 3mo agoView on Hugging Face
1likes
App README

Visual Recall

Private, AI-powered image memory for search, browsing, and retrieval.

Visual Recall turns screenshots, photos, reference images, UI captures, diagrams, documents, and visual notes into a searchable personal image library. It indexes images with a local multimodal embedding model, extracts visible text with OCR, encrypts uploaded image bytes, and stores each user's retrieval data separately.

The current deployment flow expects a persistent bucket mounted at /data.

Product Overview

Visual Recall is built for people who collect visual information and need to find it again later:

  • —Designers searching UI screenshots, layouts, references, and product captures.
  • —Developers indexing diagrams, app states, architecture sketches, and errors.
  • —Researchers organizing visual notes, documents, figures, and evidence.
  • —Operators maintaining searchable collections of screenshots and visual records.
  • —Anyone who wants a private image memory that supports natural-language search.

Upload images once, then search for them using descriptions like:

  • —red car parked in front of a glass building
  • —whiteboard diagram with arrows and product architecture
  • —invoice screenshot with table and totals
  • —dark UI dashboard with revenue chart
  • —image containing handwritten notes

Current Flow

1. Login

Hugging Face login is required before upload and search pages are available. The Space uses native Hugging Face OAuth through README metadata:

yaml
hf_oauth: true
hf_oauth_scopes:
  - email

When the Space starts, Hugging Face provides OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_SCOPES, and SPACE_HOST environment variables. The Streamlit app uses those native Space-managed values to render a Hugging Face sign-in link, exchange the callback code, and read username/email from the OAuth userinfo endpoint. openid and profile are always included by Spaces OAuth; email is requested through hf_oauth_scopes.

2. Upload

Add one or many JPG, JPEG, or PNG images. Each image is converted to RGB, serialized as PNG bytes, hashed with the logged-in user's derived key, and checked for duplicates before indexing.

3. Encrypt

Original image bytes are encrypted with Fernet before persistence. The encrypted payload is written through encrypted_images/, which the Docker entrypoint links to /data/encrypted_images.

4. OCR

Tesseract extracts visible English text from each image. The extracted text is stored as image_ocr metadata and is used for BM25 keyword ranking during search.

5. Embed

The app loads google/siglip2-base-patch16-512 with Hugging Face Transformers. Image uploads are embedded with SigLIP image features. Search queries are embedded with SigLIP text features. Embeddings are normalized and stored in ChromaDB using cosine distance.

The model is loaded from /data/models/siglip2-base-patch16-512 when present. If it is missing, the app downloads it from Hugging Face and saves it there.

6. Search

Search combines two retrieval signals:

  • —SigLIP vector similarity finds images whose visual embedding matches the query.
  • —BM25 keyword ranking rewards exact overlap with OCR text found in candidate images.

The Search page lets you tune similarity and BM25 weights from the sidebar.

Architecture History

The first implementation used a caption-first retrieval architecture:

  1. 1.A user uploaded an image.
  2. 2.The app sent the image to a selected vision model.
  3. 3.The vision model generated a detailed textual description.
  4. 4.Ollama embeddinggemma:latest embedded that generated description.
  5. 5.ChromaDB stored the text embedding, description metadata, and encrypted image reference.
  6. 6.Search embedded the user query with the same text embedding model and blended vector similarity with BM25 over generated descriptions.

That design worked, but ingestion latency was dominated by image captioning:

  • —Local vision model ingestion took about 60 seconds per image.
  • —Cloud vision model ingestion took about 16 seconds per image.

The current implementation removes image caption generation from the ingestion path. It uses a local SigLIP multimodal model directly: uploaded images are embedded with image features, search queries are embedded with text features, and Tesseract OCR supplies exact text for BM25.

Current ingestion takes about 2 to 4 seconds per image using local models alone.

Compared with the old local-model path, this is:

  • —93.3% to 96.7% less ingestion time.
  • —15x to 30x higher ingestion throughput.
  • —1,400% to 2,900% faster by throughput increase.

Compared with the old cloud-model path, this is:

  • —75.0% to 87.5% less ingestion time.
  • —4x to 8x higher ingestion throughput.
  • —300% to 700% faster by throughput increase.

Core Features

  • —Streamlit interface with Home, Upload Image, and Search Image pages.
  • —Hugging Face-login gated upload and search.
  • —Multi-image upload for common web image formats.
  • —Duplicate detection using a content-based hash tied to the user key.
  • —Local SigLIP image and text embeddings through Hugging Face Transformers.
  • —Tesseract OCR for visible text extraction.
  • —ChromaDB persistent per-user vector storage.
  • —Fernet encryption for saved image content.
  • —Gallery browsing with pagination.
  • —Hybrid vector similarity plus BM25 search ranking.
  • —Sidebar controls for result count, similarity weight, and BM25 weight.
  • —Docker deployment path for Hugging Face Spaces.

Technical Architecture

Visual Recall uses a practical local-first architecture:

  • —Frontend: Streamlit
  • —Authentication: Native Hugging Face Spaces OAuth app variables
  • —Image processing: Pillow and NumPy
  • —OCR: Tesseract via pytesseract
  • —Embeddings: google/siglip2-base-patch16-512
  • —Model runtime: Hugging Face Transformers and PyTorch
  • —Vector database: ChromaDB
  • —Keyword ranking: BM25
  • —Encryption: Fernet symmetric encryption
  • —Persistent storage: mounted /data bucket
  • —Deployment: Docker, designed for Hugging Face Spaces

Upload Pipeline

  1. 1.Streamlit accepts one or more image uploads.
  2. 2.Pillow converts each image to RGB and serializes it as PNG bytes.
  3. 3.A SHA-256 id is created from the image bytes plus the user's derived key.
  4. 4.ChromaDB checks for an existing id and skips duplicates.
  5. 5.Fernet encrypts the original image bytes.
  6. 6.The encrypted payload is saved under /data/encrypted_images.
  7. 7.Tesseract extracts English OCR text from the image.
  8. 8.SigLIP creates a normalized image embedding.
  9. 9.ChromaDB stores the embedding, metadata, OCR text, and image reference under a hashed user directory in /data/dbs.

Search Pipeline

  1. 1.The user enters a natural-language query.
  2. 2.SigLIP creates a normalized text embedding for the query.
  3. 3.ChromaDB retrieves a candidate set using cosine similarity.
  4. 4.Candidate OCR text is tokenized for BM25 ranking.
  5. 5.Similarity and BM25 scores are normalized.
  6. 6.The app combines scores using the sidebar weights.
  7. 7.Matching encrypted images are decrypted only when displayed.

Storage Model

Runtime data lives in the mounted /data bucket:

  • —/data/dbs/<hashed user id>/ contains per-user ChromaDB collections.
  • —/data/encrypted_images/ contains encrypted image payloads.
  • —/data/models/siglip2-base-patch16-512/ contains the cached SigLIP model.

Inside the app container, encrypted_images/ is a symlink to /data/encrypted_images so the current application paths remain stable.

The repository should not track downloaded model files under data/models. Model files are runtime/cache data and should be provided by the mounted bucket or downloaded on first use.

Running Locally

Install Python dependencies:

bash
pip install -r requirements.txt

Install system OCR dependencies if they are not already available:

bash
sudo apt-get update
sudo apt-get install -y tesseract-ocr tesseract-ocr-eng

Create the expected runtime directories:

bash
mkdir -p /data/dbs /data/encrypted_images /data/models

For local development outside Hugging Face Spaces, set a mock user before starting Streamlit:

bash
export HF_USER_NAME="local-dev"
export HF_USER_EMAIL="local-dev@example.com"

Then start the app:

bash
streamlit run Homepage.py

Running With Docker

Build the image:

bash
docker build -t visual-recall .

Run the container with a persistent bucket mounted at /data:

bash
docker run --rm \
  -p 8501:8501 \
  -v visual-recall-data:/data \
  -e HF_USER_NAME="local-dev" \
  -e HF_USER_EMAIL="local-dev@example.com" \
  visual-recall

The Docker image installs Tesseract binaries and English OCR data. The entrypoint also checks for tesseract at startup and installs it as a fallback if it is not present in the runtime environment.

At startup, the entrypoint:

  • —Creates /data, /data/dbs, /data/models, and /data/encrypted_images.
  • —Seeds bundled model files from /app/data/models into /data/models if the mounted model directory is empty.
  • —Links /app/encrypted_images to /data/encrypted_images.
  • —Starts Ollama and pulls embeddinggemma:latest for compatibility with the current container setup.
  • —Launches Streamlit on port 8501.

Hugging Face Spaces

This repository is configured for Hugging Face Docker Spaces:

  • —sdk: docker
  • —app_port: 8501
  • —hf_oauth: true
  • —hf_oauth_scopes: [email]
  • —Streamlit binds to 0.0.0.0
  • —A persistent Space storage bucket should be mounted at /data
  • —The app uses Hugging Face's Space-managed OAuth environment variables

Because the SigLIP model can be large, the first startup or first upload may take longer if /data/models/siglip2-base-patch16-512 is not already populated.

Security Notes

Visual Recall is designed as a local-first prototype with practical privacy controls:

  • —Hugging Face login gates access to upload and search pages.
  • —ChromaDB paths are separated by a stable hash of the Hugging Face user claims.
  • —Original image content is encrypted before persistence.
  • —Images are decrypted only when they need to be displayed.
  • —Searchable metadata, OCR text, embeddings, and image arrays are stored in ChromaDB.
  • —Local development can use HF_USER_NAME and HF_USER_EMAIL as mock headers.

For production use, harden user identity, key derivation, key rotation, access controls, retention policies, backups, and storage isolation.

Recommended First Run

  1. 1.Confirm the Space metadata includes hf_oauth: true.
  2. 2.Start the app and log in with the in-app Hugging Face sign-in button.
  3. 3.Upload 5 to 10 varied images.
  4. 4.Search for broad concepts such as dashboard, receipt, or street scene.
  5. 5.Search for narrow details such as a visible word, color, or object.
  6. 6.Adjust similarity and BM25 weights to compare ranking behavior.
  7. 7.Upload the rest of your library in batches once results look useful.

Roadmap

Planned extensions include:

  • —Video upload and frame extraction.
  • —Audio transcription for searchable video context.
  • —Timestamp-level retrieval for video results.
  • —Stronger query/result explainability.
  • —More explicit ranking controls and result filters.
  • —Production-grade authentication, key management, and storage isolation.