su00mit0001/newsgroups-semantic-search

Newsgroups Semantic Search

Lightweight semantic search over the 20 Newsgroups dataset with fuzzy clustering and a semantic cache layer.

Architecture

ingest.py     → cleans corpus → embeds with SentenceTransformers → persists in ChromaDB
cluster.py    → UMAP dimensionality reduction → GMM fuzzy clustering → saves models
main.py       → FastAPI service wiring everything together
cache.py      → SemanticCache class (from scratch, no Redis)

Setup

# 1. Create virtual environment
python -m venv venv
source venv/bin/activate        # Windows: venv\Scripts\activate

# 2. Install dependencies
pip install -r requirements.txt

# 3. Embed the corpus (run once — takes ~5 mins on CPU)
python ingest.py

# 4. Fit clusters (run once — takes ~10 mins on CPU)
python cluster.py

# 5. Start the API
uvicorn main:app --reload --port 8000

API docs available at: http://localhost:8000/docs

Endpoints

POST /query

curl -X POST http://localhost:8000/query \
  -H "Content-Type: application/json" \
  -d '{"query": "what are the best firearms for home defense?"}'

Response:

{
  "query": "what are the best firearms for home defense?",
  "cache_hit": false,
  "matched_query": null,
  "similarity_score": null,
  "result": "[Result 1] Category: talk.politics.guns | ...",
  "dominant_cluster": 4
}

GET /cache/stats

curl http://localhost:8000/cache/stats

DELETE /cache

curl -X DELETE http://localhost:8000/cache

Design Decisions

Part 1 — Embeddings

• —Model: all-MiniLM-L6-v2 — 384-dim, fast on CPU, strong semantic quality
• —Cleaning: remove=('headers', 'footers', 'quotes') — strips metadata noise
• —Vector DB: ChromaDB with cosine distance index

Part 2 — Fuzzy Clustering

• —Why GMM not K-Means: GMM gives a probability distribution per document. A post about gun legislation belongs to both politics and firearms — GMM captures this, K-Means cannot.
• —Why UMAP before GMM: 384-dim space is too sparse for Gaussian blobs. UMAP reduces to 20 dims while preserving semantic neighbourhood structure.
• —K selection: BIC curve (see bic_curve.png after running cluster.py)

Part 3 — Semantic Cache

• —Data structure: dict[cluster_id → list[entries]] — cluster-bucketed for O(n/k) lookup
• —Similarity: cosine similarity from scratch (3 lines of NumPy)
• —Threshold τ = 0.90 by default — empirically chosen after exploring:
• —τ=0.95 → misses obvious synonyms (too strict)
• —τ=0.90 → same concept, different words → hit (good default)
• —τ=0.80 → loosely related queries hit (too permissive)
• —τ=0.70 → different topics start hitting (broken)

Docker

# Build
docker build -t newsgroups-search .

# Run
docker run -p 8000:8000 newsgroups-search

Deployment Notes

Why large files are not in the repo

The following files are generated locally and excluded from the repo due to size:

To run locally from scratch

# 1. Clone the repo
git clone https://github.com/YOUR_USERNAME/newsgroups-semantic-search.git
cd newsgroups-semantic-search

# 2. Set up venv
python -m venv venv
venv\Scripts\activate        # Windows
source venv/bin/activate     # Mac/Linux

# 3. Install dependencies
pip install -r requirements.txt

# 4. Download the dataset
# https://archive.uci.edu/dataset/113/twenty+newsgroups
# Extract and place the 20_newsgroups folder in the project root
# Update DATASET_PATH in ingest.py if needed

# 5. Generate embeddings and models (run once)
python ingest.py     # ~5 mins
python cluster.py    # ~10 mins

# 6. Start the API
uvicorn main:app --reload --port 8000

Live API

The live deployment is available at: [deployed URL]

Note: The service requires the generated model files (umap_reducer.pkl, gmm.pkl, chroma_db/) to be present. These are generated by running ingest.py and cluster.py and are excluded from version control due to size constraints. For the live deployment, these files were generated on the server before starting the service.