Apppublic

imkk21/custom-vector-search

sourceHugging Faceupdated 3mo agoView on Hugging Face

0likes

App README

Custom HNSW Vector Search Engine from Scratch

A high-performance Hierarchical Navigable Small World (HNSW) vector search engine written from scratch in Python using NumPy. It includes custom implementations of L2/Cosine distance metrics, an HNSW graph traversal algorithm, a FastAPI web service, and a benchmarking suite to compare performance against Facebook's FAISS library.

Architecture Overview

HNSW is a graph-based Approximate Nearest Neighbor (ANN) search algorithm. It builds a multi-layer graph where:

—Upper layers act as "highways" with sparse node connections, allowing fast greedy navigation across large vector distances.
—Lower layers (down to Level 0) increase connection density, allowing local, high-precision search.

Project Structure

—src/metrics.py - Custom L2 (Euclidean) and Cosine distance calculations.
—src/hnsw.py - Core HNSW logic, including Node, HNSWGraph, multi-layer insertion, and k-NN query.
—src/embedder.py - Integrates sentence-transformers (all-MiniLM-L6-v2) to convert text to 384-dimensional vectors.
—src/app.py - FastAPI app wrapping the engine with /index and /search REST endpoints.
—tests/test_hnsw.py - Pytest suite testing graph search correctness.
—tests/benchmark.py - Performance comparison script checking accuracy (Recall) and speed (QPS) vs FAISS.

Setup & Running

1. Install Dependencies

bash

pip install -r requirements.txt

2. Run Tests

Validate distance metrics and search accuracy:

bash

python -m pytest tests/test_hnsw.py

3. Run Benchmarks

Compare your custom graph against FAISS:

bash

python -m tests.benchmark

This generates a performance comparison chart saved in `visualizations/benchmark.png`.

4. Start FastAPI Server

Deploy the REST service locally:

bash

uvicorn src.app:app --reload

Once running, navigate to http://127.0.0.1:8000/docs to test document indexing and semantic search queries via the Swagger UI.