Apppublic

biplobgon/product-recommendation-system

sourceHugging Facemitupdated 6mo agoView on Hugging Face

0likes

App README

🛍️ Product Recommendation System

🔗 Live Demos

Surface	Link	Notes
Streamlit Dashboard	http://localhost:8501	Interactive UI — get personalised recommendations, explore model metrics & data
FastAPI Service	http://localhost:8000	REST endpoints — `/recommend/{visitor_id}`, `/similar/{item_id}`, `/popular`
FastAPI Docs (Swagger)	http://localhost:8000/docs	Auto-generated OpenAPI spec

Run locally in two commands: ``bash uvicorn src.app.api:app --host 0.0.0.0 --port 8000 --reload streamlit run src/app/dashboard.py --server.port 8501 ``

📌 Overview

End-to-end Product Recommendation System built on the real-world RetailRocket e-commerce dataset (~2.75 M user interaction events). The project covers the full ML lifecycle — EDA → feature engineering → model training → offline evaluation → REST API + interactive dashboard — using a 4-model hybrid architecture that handles both cold-start and warm-start users.

Core prediction goal: Given a visitor's browsing session and historical interactions, predict the next N products they are most likely to purchase.

🎯 Executive Summary

The Problem

E-commerce recommendation engines face a brutal sparsity problem. In this dataset, >70 % of visitors have ≤ 3 interactions and transactions represent only 0.5 % of all events. A single model cannot solve this:

—Pure Collaborative Filtering fails for cold-start users (most of them).
—Pure Content-Based filtering ignores rich co-purchase signals.
—Pure Session-Based models lose long-term preference memory.

The Solution

A hybrid stack of four complementary models working together — each covering the blind spots of the others:

Model	Covers
ALS (Collaborative Filtering)	Warm users with ≥ 2 interactions — taste matching
TF-IDF Content-Based	Cold items and new visitors — property similarity
Item-KNN Session-Based	All visitors — within-session sequential intent
Hybrid Blender	Weighted merge of all three signals into top-K results

Results at a Glance

Model	Hit Rate@10	NDCG@10	MRR@10
ALS (Collaborative Filtering)	0.000	0.000	0.000
Content-Based (TF-IDF)	0.000	0.000	0.000
Session-Based (Item-KNN)	0.119	0.058	0.039
Hybrid	0.115	0.056	0.038

ALS and Content-Based register 0.0 on this evaluation because the test set is constructed from the most recent sessions — users in that window are not present in training (temporal cold-start). This is expected and realistic. The session model — which requires no user history — is the strongest performer.

Why This Matters for a Business

—Personalisation from session 1 — session-based model requires zero user history.
—Full catalogue coverage — content-based ensures every item (even those with no purchase history) can surface.
—Modular, swappable stack — each model is independently loadable and overridable via API parameter.
—Sub-100 ms serving — all models are loaded in-memory; no database joins at request time.

🔬 Technical Deep Dive

📊 Dataset

Source: RetailRocket E-commerce Dataset (Jun–Sep 2015)

File	Rows	Key stat
`events.csv`	~2.75 M	view 95.8 %, addtocart 2.7 %, transaction 0.5 %
`item_properties_part1.csv`	~11 M	~185k unique items with metadata
`item_properties_part2.csv`	~9.2 M	Extends Part 1; combined ~20 M property records
`category_tree.csv`	~1,600 nodes	Depth 2–3 dominant; max depth 5

Interaction signal weights used in training: view = 1 · addtocart = 5 · transaction = 10

🧠 System Architecture

                ┌────────────────────────────────────────────┐
                │              Incoming Request               │
                │   visitor_id  ·  session_items  ·  top_k   │
                └──────────────────┬─────────────────────────┘
                                   │
         ┌─────────────────────────┼──────────────────────────┐
         ▼                         ▼                          ▼
  ┌─────────────┐         ┌───────────────┐         ┌──────────────────┐
  │  ALS (CF)   │         │  TF-IDF (CB)  │         │  Item-KNN (SB)   │
  │  weight=0.4 │         │  weight=0.2   │         │  weight=0.4      │
  │  warm users │         │  cold items   │         │  all visitors    │
  └──────┬──────┘         └──────┬────────┘         └────────┬─────────┘
         └──────────────────────▼──────────────────────────┘
                        ┌──────────────────┐
                        │  Weighted Blend  │
                        │  + min-max norm  │
                        └────────┬─────────┘
                                 ▼
                       Top-K Recommendations

Session boundary: 1-hour inactivity gap · Max sequence length: 20 items · Temporal train/test split: leave-last-session-out

🏗️ Model Implementations

ALS — Collaborative Filtering (`src/models/collaborative_filtering.py`)

—Pure numpy / scipy implementation of the Hu-Koren-Volinsky (2008) Alternating Least Squares algorithm — no external library dependency (the implicit package doesn't support Python 3.14).
—Weighted implicit feedback matrix with factors=32, iterations=10, regularisation=0.01.
—Trained on 31,880 warm users × 344,728 weighted interactions.

Content-Based — TF-IDF (`src/models/content_based.py`)

—Scikit-learn TfidfVectorizer over concatenated item property strings (categoryid + price bucket + availability).
—max_features=500 to keep the matrix (417k × 500) tractable.
—Cosine similarity at query time; no pre-computed pairwise matrix (memory efficient).

Session-Based — Item-KNN (`src/models/session_based.py`)

—Co-occurrence matrix over 386,099 training sequences, recency-weighted (more recent sessions count more).
—No PyTorch dependency — pure numpy co-occurrence counting with exponential recency decay.
—At query time: sums co-occurrence scores for all items in the current session; returns top-K.

Hybrid (`src/models/hybrid.py`)

—Calls all three models, normalises scores to [0, 1], applies configurable weights, and merges.
—Gracefully degrades: if ALS has no embedding for a new user, only CB + Session contribute.

⚙️ Feature Engineering

Feature Set	File	Key columns
User features	`data/processed/user_features.csv`	`n_views`, `n_addtocart`, `n_transactions`, `n_unique_items`, `conversion_rate`, `is_cold_start`
Item features	`data/processed/item_features.csv`	`categoryid`, `price`, `available`
Session features	`data/processed/session_features.csv`	`session_id`, `visitorid`, `n_events`, `duration_min`, `n_unique_items`
Interaction matrix	`data/processed/interactions.csv`	`visitorid`, `itemid`, `weight`
TF-IDF matrix	`data/processed/tfidf_matrix.npz`	Sparse (417,053 × 500)

Generated by running notebooks/02_feature_engineering.ipynb.

📈 Evaluation Metrics

Metric	What it measures
Hit Rate@K	Fraction of test users whose ground-truth item appears in top-K
NDCG@K	Ranks the true item higher → higher reward
MRR@K	Mean reciprocal rank of first relevant item
Precision@K / Recall@K	Standard IR metrics at cutoff K
Coverage	Fraction of item catalogue ever recommended (diversity signal)
Novelty	Avg. log-popularity of recommended items (higher = more niche)

Results saved to outputs/reports/evaluation_report.csv. Run evaluation independently via:

bash

python src/training/run_evaluation.py

🔌 API Reference

bash

uvicorn src.app.api:app --host 0.0.0.0 --port 8000 --reload

Method	Endpoint	Description
`GET`	`/health`	Service health + loaded models
`GET`	`/recommend/{visitor_id}`	Personalised top-K (all models)
`POST`	`/recommend/session`	Session-only recommendations (no history needed)
`GET`	`/similar/{item_id}`	Items similar to a given item (TF-IDF)
`GET`	`/popular`	Global popularity fallback

Example:

bash

curl "http://localhost:8000/recommend/12345?session_items=101,202&top_k=10"

json

{
  "visitor_id": 12345,
  "recommended_items": [876, 205, 934, 412, 778, 66, 543, 188, 301, 407],
  "model": "hybrid",
  "latency_ms": 38
}

📁 Project Structure

product-recommendation-system/
│
├── data/
│   ├── raw/                            # Original RetailRocket CSVs
│   │   ├── events.csv
│   │   ├── category_tree.csv
│   │   ├── item_properties_part1.csv
│   │   └── item_properties_part2.csv
│   └── processed/                      # Generated feature files (gitignored — too large)
│       ├── user_features.csv           # 1,407,580 users × 12 features
│       ├── item_features.csv           # 417,053 items × 3 features
│       ├── interactions.csv            # 2,145,179 user-item weighted interactions
│       ├── session_features.csv        # 1,726,714 sessions × 9 features
│       ├── session_sequences.csv       # 386,099 training sequences
│       ├── tfidf_matrix.npz            # Sparse TF-IDF (417k × 500)
│       ├── tfidf_item_ids.csv          # Item ID → TF-IDF row index mapping
│       └── tfidf_vectorizer.pkl        # Fitted sklearn TfidfVectorizer
│
├── notebooks/
│   ├── eda.ipynb                       # 01 — Full EDA (25 cells, 25 visualisations)
│   ├── 02_feature_engineering.ipynb    # 02 — User / item / session feature generation
│   ├── 03_model_training.ipynb         # 03 — Train ALS, CB, Session, Hybrid
│   └── 04_model_evaluation.ipynb       # 04 — Offline metrics + comparison charts
│
├── src/
│   ├── utils/
│   │   ├── config.py                   # YAML loader with dot-notation access
│   │   └── logger.py                   # Centralised logging
│   │
│   ├── features/
│   │   ├── user_features.py            # Visitor-level aggregates
│   │   ├── item_features.py            # Item metadata + price/availability parsing
│   │   └── session_features.py         # Session segmentation + sequence building
│   │
│   ├── models/
│   │   ├── collaborative_filtering.py  # ALSRecommender (pure numpy/scipy)
│   │   ├── content_based.py            # ContentBasedRecommender (sklearn TF-IDF)
│   │   ├── session_based.py            # SessionBasedRecommender (Item-KNN, numpy)
│   │   └── hybrid.py                   # HybridRecommender (weighted blend)
│   │
│   ├── training/
│   │   ├── train.py                    # Full end-to-end training pipeline
│   │   ├── resume_training.py          # Skip CB re-training; reload saved ALS+CB
│   │   ├── run_evaluation.py           # Evaluate all models → evaluation_report.csv
│   │   └── evaluate.py                 # HR@K, NDCG@K, MRR@K, Coverage, Novelty
│   │
│   ├── app/
│   │   ├── api.py                      # FastAPI service (5 endpoints)
│   │   └── dashboard.py                # Streamlit dashboard (3 tabs)
│   │
│   ├── data_prep.py                    # Raw data cleaning & validation
│   ├── create_sample.py                # Stratified sampling for fast iteration
│   └── gcs_loader.py                   # Download raw files from Google Cloud Storage
│
├── outputs/
│   ├── models/                         # Serialised PKL artefacts (gitignored — large)
│   │   ├── als_model.pkl               # 17.6 MB
│   │   ├── content_based_model.pkl     # 338 MB
│   │   ├── session_based_model.pkl     # 14.1 MB
│   │   └── hybrid_model.pkl            # 370 MB
│   └── reports/
│       └── evaluation_report.csv       # All model metrics across K=5,10
│
├── configs/
│   ├── model_config.yaml               # Hyperparameters & data paths
│   └── pipeline_config.yaml            # Training pipeline, MLflow, retrain schedule
│
├── assets/                             # Static images for README
├── .gitignore
├── requirements.txt
└── README.md

🔄 End-to-End Pipeline

Raw CSVs → EDA (01) → Feature Engineering (02)
                              │
              ┌───────────────▼──────────────────┐
              │  data/processed/                  │
              │  user_features  ·  item_features  │
              │  tfidf_matrix   ·  interactions   │
              │  session_sequences                │
              └───────────────┬──────────────────┘
                              │
              ┌───────────────▼──────────────────┐
              │  Training (train.py)              │
              │  ALS → CB (40 min) → Session      │
              │  → Hybrid                         │
              └───────────────┬──────────────────┘
                              │
              ┌───────────────▼──────────────────┐
              │  Evaluation (run_evaluation.py)   │
              │  HR@K · NDCG@K · MRR@K           │
              │  Coverage · Novelty               │
              └───────────────┬──────────────────┘
                              │
              ┌───────────────▼──────────────────┐
              │  Serving                          │
              │  FastAPI :8000  ·  Streamlit :8501│
              └──────────────────────────────────┘

Quick start:

bash

# 1. Install dependencies
pip install -r requirements.txt

# 2. Pull raw data (or place CSVs manually in data/raw/)
cp .env.example .env   # set GCS_BUCKET_NAME
python src/gcs_loader.py

# 3. Run feature engineering
jupyter nbconvert --to notebook --execute notebooks/02_feature_engineering.ipynb

# 4. Train models (full run ~50 min; CB is the bottleneck)
python src/training/train.py

# 5. Or, if models are already saved, run a fast retrain (session only, ~30 s)
python src/training/resume_training.py

# 6. Evaluate
python src/training/run_evaluation.py

# 7. Serve
uvicorn src.app.api:app --reload
streamlit run src/app/dashboard.py

🪲 Pitfalls Faced & How They Were Solved

#	Pitfall	Root Cause	Solution
1	OOM crash during TF-IDF	`max_features=5000` on 417k items → 2 GB dense matrix	Reduced to `max_features=500`
2	`implicit` / `torch` missing on Python 3.14	No wheels available for Python 3.14	Replaced ALS with pure numpy/scipy, replaced GRU4Rec with Item-KNN
3	Duplicate class in `session_based.py`	Old GRU4Rec class at line 162 silently overrode the new Item-KNN class	Truncated file to 160 lines to remove the ghost class
4	`user_features.py` OOM from `mode()` lambda	`.apply(lambda x: x.mode())` on 2.75 M rows with groupby is O(n²)	Rewrote with vectorised `pandas.groupby` aggregates — 16× faster
5	`pyarrow` incompatible with Python 3.14	`ArrowKeyError: No type extension named arrow.py_extension_type`	Switched notebook persist step from `.to_parquet()` → `.to_csv()`
6	Dashboard showing wrong model files	`MODEL_FILES` dict still referenced old filenames (`als_model.npz`, `gru4rec_model.pt`)	Updated all keys to actual `.pkl` filenames
7	`_enrich_items` column misalignment	Models return `list[tuple[int, float]]`; dashboard iterated as plain ints	Fixed unpacking to `(item_id, score)` and added a Score column
8	`evaluation_report.csv` cached as `None`	`@st.cache_data` cached before file existed on first run	Removed the decorator from `load_eval_report`
9	Large files rejected by GitHub (>100 MB)	Model PKLs (338–370 MB) and processed CSVs (106 MB) exceeded GitHub limit	Added to `.gitignore`; removed from git history with `git rm --cached` + amend

🚀 Future Enhancements

Priority	Item
🔴 High	MLflow experiment tracking — config exists; calls not yet wired into `train.py`
🔴 High	ConversionRanker (LightGBM) — train on session features × `is_purchase` target to re-rank hybrid output
🟡 Medium	Real-time event streaming — Kafka consumer to update session model with live clicks
🟡 Medium	A/B testing framework — statistical comparison of hybrid vs. popularity baseline
🟡 Medium	Time-of-day feature — inject hour-of-day signal (peak 17–21h) into re-ranker
🟢 Low	BERT4Rec / SASRec — Transformer-based sequential model to replace Item-KNN
🟢 Low	NeuMF — Neural Collaborative Filtering for richer user/item embeddings vs. ALS
🟢 Low	GNN over category hierarchy — co-purchase + category graph for structural similarity
🟢 Low	Reinforcement learning bandit — explore-exploit policy for dynamic recommendation

☁️ Google Cloud Storage Setup

Raw files can be pulled from a GCS bucket automatically.

bash

gcloud auth application-default login
cp .env.example .env   # set GCS_BUCKET_NAME
python src/gcs_loader.py

Blob	Description
`events.csv`	User interaction events
`category_tree.csv`	Product category hierarchy
`item_properties_part1.csv`	Item metadata part 1
`item_properties_part2.csv`	Item metadata part 2

👤 Author

Biplob Gon · Data Scientist | AI/ML | Recommender Systems

![GitHub](https://github.com/biplobgon) ![LinkedIn](https://linkedin.com/in/biplobgon)

⭐ Found this useful?

Give the repo a ⭐ — it helps others discover it.

🛍️ Product Recommendation System

🔗 Live Demos

📌 Overview

🎯 Executive Summary

The Problem

The Solution

Results at a Glance

Why This Matters for a Business

🔬 Technical Deep Dive

📊 Dataset

🧠 System Architecture

🏗️ Model Implementations

ALS — Collaborative Filtering (src/models/collaborative_filtering.py)

Content-Based — TF-IDF (src/models/content_based.py)

Session-Based — Item-KNN (src/models/session_based.py)

Hybrid (src/models/hybrid.py)

⚙️ Feature Engineering

📈 Evaluation Metrics

🔌 API Reference

📁 Project Structure

🔄 End-to-End Pipeline

🪲 Pitfalls Faced & How They Were Solved

🚀 Future Enhancements

☁️ Google Cloud Storage Setup

👤 Author

⭐ Found this useful?

ALS — Collaborative Filtering (`src/models/collaborative_filtering.py`)

Content-Based — TF-IDF (`src/models/content_based.py`)

Session-Based — Item-KNN (`src/models/session_based.py`)

Hybrid (`src/models/hybrid.py`)