ereztobias/CatalogueSearch

CatalogueSearch with ChromaDB + RAG

AI-powered semantic search for catalogues using ChromaDB, vector embeddings, and RAG (Retrieval-Augmented Generation) with Claude AI.

Features

• —Semantic Search: ChromaDB vector database with 26,818+ clothing items
• —Smart Filtering: Automatic extraction of gender, category, color, and stock filters
• —RAG Mode: AI-powered recommendations using Claude API
• —Web UI: Flask-based visual search interface
• —Image Cache: Separate JSON cache for product images
• —Persistent Storage: Local vector database with metadata filtering
• —50% Match Threshold: Only shows highly relevant results

Installation

1. 1.Install dependencies:

pip install -r requirements.txt

1. 1.Set up environment variables (for RAG mode):

# Copy the example file
cp .env.example .env

# Edit .env and add your Anthropic API key
# Get your key from: https://console.anthropic.com/
ANTHROPIC_API_KEY=your-key-here

Quick Start

Load the Catalogue

First, load the clothing catalogue into ChromaDB:

python load_by_category.py

This will fetch and embed 26,818 products across 5 categories:

• —Shirts (16,625 items)
• —Pants (5,518 items)
• —Dresses (2,606 items)
• —Jeans (1,400 items)
• —Skirts (669 items)

Build Image Cache (Optional)

Build the image cache for the web UI:

python build_image_cache.py

Run the Web UI

Launch the Flask web interface:

python browse_ui.py

Then open http://localhost:5000 in your browser.

Features:

• —Standard semantic search
• —RAG mode with AI recommendations (toggle checkbox)
• —Product images and details
• —Filter detection (gender, category, color, stock)
• —Match percentage display

Search Modes

1. Semantic Search (Default)

Pure vector similarity search with smart filtering:

python smart_search.py

Example queries:

• —"blue jeans for women"
• —"casual shirts for men"
• —"summer dress available in stock"

2. RAG Mode (AI-Powered)

Retrieval + AI-generated recommendations:

python rag_search.py

How it works:

1. 1.Retrieves relevant products using semantic search
2. 2.Sends top matches to Claude API
3. 3.Generates personalized, conversational recommendations
4. 4.Returns both AI recommendation AND product results

Example query → AI response:

"pants for a conference for women" "For a professional conference setting, I'd recommend these tailored options... The gray suit trousers from Brand X offer excellent quality at £89, currently on sale from £125..."

Basic Usage

from chroma_search import CatalogueSearch

# Initialize search
search = CatalogueSearch()

# Add items to catalogue
items = [
    {
        "id": "prod_001",
        "text": "Premium wireless Bluetooth headphones with noise cancellation",
        "category": "electronics",
        "price": "199.99"
    },
    {
        "id": "prod_002",
        "text": "Organic cotton t-shirt, eco-friendly",
        "category": "clothing",
        "price": "29.99"
    }
]
search.add_items(items)

# Search the catalogue
results = search.search("headphones for music", n_results=5)
for doc, distance in zip(results["documents"], results["distances"]):
    print(f"{doc} (similarity: {1 - distance:.3f})")

Run the Example

python chroma_search.py

This will:

1. 1.Create a ChromaDB collection
2. 2.Add sample catalogue items
3. 3.Demonstrate semantic search with different queries
4. 4.Show how to use metadata filters

API Reference

CatalogueSearch

__init__(collection_name, persist_directory)

Initialize the ChromaDB client and collection.

• —collection_name (str): Name for the collection (default: "catalogue_items")
• —persist_directory (str): Directory to persist data (default: "./chroma_db")

add_items(items)

Add items to the catalogue.

• —items (List[Dict]): List of items with 'id', 'text', and optional metadata

search(query, n_results, filter_metadata)

Search the catalogue using semantic similarity.

• —query (str): Search query text
• —n_results (int): Number of results to return (default: 5)
• —filter_metadata (Dict): Optional metadata filter

Returns a dictionary with:

• —ids: List of matching item IDs
• —documents: List of matching documents
• —distances: List of cosine distances (lower = more similar)
• —metadatas: List of metadata dictionaries

delete_items(ids)

Delete items by ID.

• —ids (List[str]): List of item IDs to delete

clear_collection()

Clear all items from the collection.

get_collection_stats()

Get collection statistics.

Advanced Usage

Metadata Filtering

# Search only within a specific category
results = search.search(
    "gadgets",
    n_results=5,
    filter_metadata={"category": "electronics"}
)

Custom Collection

# Use a custom collection name and storage location
search = CatalogueSearch(
    collection_name="my_products",
    persist_directory="./my_db"
)

How It Works

1. 1.Vector Embeddings: Text descriptions are automatically converted to vector embeddings using sentence transformers
2. 2.Similarity Search: Queries are embedded and compared using cosine similarity
3. 3.Persistence: All data is stored locally in the ChromaDB directory
4. 4.Semantic Understanding: The system understands meaning, not just keywords (e.g., "headphones" matches "audio equipment")

Data Structure

Each catalogue item should have:

• —id (required): Unique identifier
• —text (required): Description or content to search
• —Additional fields: Any other metadata (category, price, etc.)

Performance Tips

• —Add items in batches for better performance
• —Use metadata filters to narrow search scope
• —The first query may be slower as models load
• —Subsequent queries are much faster

Architecture

Semantic Search Flow

1. 1.Query → Extract filters (gender, category, color, stock)
2. 2.Clean query (remove filter keywords)
3. 3.Embed query with all-MiniLM-L6-v2 (384 dimensions)
4. 4.Search ChromaDB with cosine similarity
5. 5.Apply metadata filters
6. 6.Filter results (≥50% match)
7. 7.Return products

RAG Flow

1. 1.Retrieval: Semantic search (above)
2. 2.Augmentation: Format top 5 products as context
3. 3.Generation: Send to Claude API with system prompt
4. 4.Output: Natural language recommendation + product results

Matching Score

Match % = (1 - cosine_distance) × 100

• —90-100%: Very strong semantic match
• —70-90%: Good semantic match
• —50-70%: Moderate match
• —<50%: Filtered out (not shown)

Project Structure

CatalogueSearch/
├── chroma_search.py         # Core ChromaDB wrapper
├── smart_search.py          # Semantic search + auto-filtering
├── rag_search.py            # RAG with Claude API
├── load_by_category.py      # Load 26K+ products from API
├── build_image_cache.py     # Build image cache
├── browse_ui.py             # Flask web UI
├── api_helper.py            # API utilities
├── search_demo.py           # CLI demo
├── chroma_db/               # Vector database (persistent)
├── image_cache.json         # Product images cache
├── .env                     # API keys (not in git)
└── requirements.txt         # Dependencies

Dependencies

• —chromadb: Vector database
• —sentence-transformers: Text embedding models (all-MiniLM-L6-v2)
• —anthropic: Claude API client for RAG
• —flask: Web UI framework
• —requests: API calls
• —python-dotenv: Environment configuration

Persistence

Data persists in the ./chroma_db directory by default. This directory contains:

• —Vector embeddings
• —Documents
• —Metadata
• —Index structures

License

MIT