piyushrai10/amazon-product-recommendation-system

[image]

Product Recommendation System For Amazon

Executive Summary

In today’s competitive e‑commerce landscape, delivering personalized recommendations is essential for increasing customer satisfaction and driving sales. This project tackles key challenges—data sparsity, positive rating bias, and the cold start problem—by building a robust recommendation system using Amazon’s electronic product reviews dataset.

Key Outcomes

• —Best Model: The hybrid model offers the most balanced performance, with an RMSE of 0.879 and Precision@10 of 0.522. It provides the best trade‑off between accuracy, diversity, and relevance.
• —Key Insights: Data sparsity (only 78,798 interactions out of a possible 10 million) and a strong positive rating bias (82% of ratings are ≥4) skew recommendations toward popular items, limiting diversity. These issues restrict personalization, favour top‑selling products, and hinder suggestions for less common preferences.
• —Cold Start Mitigation: The hybrid model effectively handles the cold start problem by incorporating a rank‑based approach, delivering meaningful recommendations for new users and products even with sparse interaction data.

Business Impact

• —Enhanced Engagement: Personalized recommendations increase customer satisfaction and retention.
• —Increased Sales: Relevant suggestions lead to higher conversion rates and larger average order values.
• —Optimized Efficiency: Streamlined strategies improve resource allocation and overall business outcomes.

Actionable Recommendations

• —Integrate Metadata: Enrich the system with content‑based filtering using product descriptions, categories, and brands to address data sparsity and cold start challenges.
• —Expand Data Sources: Leverage implicit feedback (clicks, cart additions, etc.) to reduce reliance on popular items, thereby improving model accuracy and recommendation diversity.
• —Optimize Diversity: Use weighted sampling, coverage metrics, and novelty metrics to promote lesser‑known products and better serve underrepresented users.
• —Implement A/B Testing: Continuously refine strategies using key indicators such as CTR, AOV, and CLV.
• —Personalize Product Promotions: Design targeted marketing campaigns that mix top‑selling and niche products, balancing popularity with diversity to drive engagement and satisfaction.

Table of Contents

• —Introduction
• —Objective
• —Key Business Metrics
• —Exploratory Data Analysis
• —Data Insights
• —Models’ Performance
• —Business Recommendations
• —Repository Structure
• —Requirements
• —Installation

Introduction

In the highly competitive e‑commerce space, personalized product recommendations are crucial for improving customer experience, increasing engagement, and boosting sales. Platforms like Amazon, Walmart, and Etsy rely on advanced recommendation engines to enhance customer retention and streamline decision‑making. This project develops a recommendation system using Amazon’s product reviews dataset, focusing specifically on electronic products. The dataset includes only product ratings (no additional metadata or review text) to ensure an unbiased model‑building process.

Objective

The goal is to build a recommendation system that suggests products to Amazon customers based on their previous ratings. Using a labeled dataset of Amazon product reviews, the project extracts meaningful insights and builds several models, including:

• —Knowledge / Rank‑Based Recommendation System: Recommends popular items to all users based on predefined criteria, helping new users when no personalization is available.
• —User‑User Collaborative Filtering: Suggests products liked by users with similar tastes by identifying similarities between users.
• —Item‑Item Collaborative Filtering: Recommends items similar to those a user has already interacted with, based on item‑to‑item similarities.
• —Matrix Factorization‑Based Collaborative Filtering: Uses matrix factorization to uncover latent factors in user‑item interactions for personalized recommendations.
• —Hybrid Recommendation System: Combines multiple techniques to improve accuracy and relevance.

Key Business Metrics

Predictive Metrics

• —Root Mean Squared Error (RMSE): Measures the accuracy of predicted ratings. Lower values indicate better performance.
• —Precision@K: The fraction of relevant items among the top K recommendations; indicates how well the model recommends engaging products.
• —Recall@K: The proportion of all relevant items that appear in the top K recommendations; reflects the model’s ability to capture diverse user preferences.
• —F1 Score@K: The harmonic mean of Precision@K and Recall@K; provides a balanced view of predictive performance.

Ranking Metrics

• —Mean Reciprocal Rank (MRR): Measures ranking quality by averaging the reciprocal rank of the first relevant item in the recommendation list.
• —Mean Average Precision (MAP): Assesses average precision across all users; indicates how well relevant items are ranked.
• —Hit Rate@K: The proportion of users for whom at least one relevant item appears in the top K recommendations.

Exploratory Data Analysis

1. 1.Data Cleaning and Preprocessing: Handled missing values and ensured data integrity for accurate modeling.
2. 2.Descriptive Statistics: Summarised user behaviour, product popularity, and rating distributions.
3. 3.Univariate Analysis: Examined individual features to identify trends and patterns in user‑product interactions.
4. 4.Visualisations: Used charts and graphs to uncover insights such as rating distribution and user engagement patterns.

Data Insights

The project analysed and compared multiple recommendation models using the Amazon electronics product reviews dataset. The evaluation revealed that data sparsity, overfitting, and the dominance of popular items significantly affect model performance.

• —Data Sparsity: With only 78,798 recorded interactions out of a possible 10 million user‑product combinations, the matrix is extremely sparse. This limits the ability to capture diverse preferences and often leads to less personalised recommendations.
• —Positive Rating Bias: More than 82% of ratings are 4 or higher. This skew toward positive feedback makes it difficult for models to generalise and to discern fine‑grained differences in user preferences.
• —Right‑Skewed Interactions: Most users have between 10 and 50 interactions, and most products are rated by only 1 to 10 users. However, a small subset of users and products dominate the interactions, introducing bias during training and skewing recommendations toward those frequent users and popular products.
• —Impact on Metrics: Data sparsity and positive rating bias cause models to favour frequently and highly rated items, inflating metrics such as Precision@K, MRR, MAP, and Hit Rate@K. This leads to overfitting and reduces Recall@K because less common but relevant items are missed.
• —Cold Start Problem: New users and newly launched or rarely rated products struggle to gain visibility because the models rely heavily on past interactions, exacerbating the cold start issue.

Models’ Performance

The following models were evaluated to determine the best approach for product recommendations:

1. 1.Rank‑Based Recommendation System
2. 2.Performance: Achieved moderate accuracy (RMSE = 0.899) and moderate Precision@K (0.565). Ranking metrics were strong (MRR = 0.842, MAP = 0.791) due to the use of Bayesian average for cold start problems.
3. 3.Insight: Effective at recommending popular items but prone to favouring frequently rated products, which limits diversity.

1. 1.User‑User Collaborative Filtering
2. 2.Performance: Showed relatively stable precision and recall after tuning, with a test RMSE of 0.888. Ranking metrics were inflated because the model tends to recommend popular items.
3. 3.Insight: Can identify user similarities but struggles with sparsity and overfitting to active user preferences.

1. 1.Item‑Item Collaborative Filtering
2. 2.Performance: Similar to user‑user CF, with a test RMSE of 0.888. The model favours popular items, leading to strong ranking metrics but limited improvement in prediction accuracy.
3. 3.Insight: Suitable for recommending similar products, but heavily influenced by frequently interacted items, which reduces its ability to capture niche preferences.

1. 1.Matrix Factorization‑Based Collaborative Filtering (SVD)
2. 2.Performance: Achieved a better balance between recall and precision with an RMSE of 0.883. Ranking metrics remained high, but the model still favoured well‑rated products, limiting generalisation.
3. 3.Insight: Better at capturing latent factors between users and items, yet still constrained by data sparsity and rating biases.

1. 1.Hybrid Recommendation System
2. 2.Performance: The most balanced model, with the lowest RMSE (0.879) and strong ranking metrics (MRR = 0.819, MAP = 0.763). By incorporating a rank‑based approach, it effectively mitigates the cold start problem, providing meaningful recommendations even for new users.
3. 3.Insight: Successfully combines the strengths of rank‑based and collaborative filtering methods, enhancing both accuracy and relevance. However, it remains influenced by data sparsity and the tendency to favour popular items.

[image]

Key Takeaways

• —★ Dominance of Popular Items: Due to data sparsity, all models tend to favour frequently rated products, which inflates ranking metrics such as MRR and MAP.
• —⚠️ Challenges: Overfitting and data sparsity remain major obstacles; models struggle to capture diverse user preferences and often focus on popular items.
• —🏆 Best Model: The hybrid model provides the most balanced performance, offering the best trade‑off between accuracy, diversity, and relevance.

Business Recommendations

Enhancing the recommendation system is expected to boost customer engagement, conversion rates, and retention, driving overall business growth. Monitoring key performance metrics is essential for assessing impact and guiding optimisation.

1. 1.Diversifying Recommendation Strategies

• —Content and Metadata Integration: Implement content‑based filtering by analysing product features and metadata (e.g., descriptions, brands, categories) to recommend items that align with user interests. This addresses both the cold start problem and data sparsity.
• —Address Data Sparsity: Use data augmentation (e.g., synthetic interactions, active learning) and implicit feedback (e.g., clicks, add‑to‑cart) to enrich the dataset and reduce reliance on popular items, promoting more diverse and accurate recommendations.
• —Continuous Model Tuning: Regularly fine‑tune model parameters to adapt to changing data patterns, improving robustness and recommendation quality.
• —Metric Reevaluation: Introduce coverage and novelty metrics to evaluate how well the model generates diverse recommendations, especially in sparse datasets. Use weighted sampling during training to prioritise underrepresented items and users, ensuring a more balanced and inclusive model.

1. 1.Strategic Business Actions

• —A/B Testing: Test different recommendation strategies (e.g., personalised vs. trending) and use key metrics—Click‑Through Rate (CTR), Conversion Rates, Average Order Value (AOV), and Customer Lifetime Value (CLV)—to optimise performance.
• —Personalise Marketing Campaigns: Tailor marketing efforts (e.g., email campaigns) based on user behaviour. Monitor CTR and Conversion Rates to refine strategies.
• —Target Low‑Engagement Segments: Identify and re‑engage less active users with tailored recommendations to reduce Churn Rate and enhance CLV.
• —Diversify Product Promotion: Promote lesser‑known products alongside top sellers using a novelty factor. Track sales growth and AOV to assess the impact on customer engagement and satisfaction.

Repository Structure

├── LICENSE <- Project's open-source license details. ├── README.md <- Top-level README for developers. │ ├── requirements.txt <- Python dependencies for replicating the environment. ├── environment.yml <- Conda environment configuration with dependencies. │ ├── data │ ├── processed <- The final, processed data sets for modeling. │ └── raw <- The original, immutable data. │ ├── models <- The final, trained and tuned recommendation model. │ ├── notebooks <- Jupyter notebooks for data exploration and analysis. │ ├── references <- Documentation, data dictionaries, and manuals. │ ├── reports <- Generated analysis as HTML, PDF, LaTeX, etc. │ └── figures <- Graphics and figures for reports. │ ├── src <- Source code for the project.

text

Requirements

Python 3.11.6 or higher is required. Download the latest version from python.org.

Installation

1. Clone the Repository

git clone https://github.com/tmoesl/product-recommendation-system.git

1. 1.Navigate to the Project Directory

cd product-recommendation-system

1. 1.Create a Virtual Environment and Install Dependencies Using conda:

conda env create -f environment.yml
conda activate product-recommendation-system-env

Using venv:

python3.12.3 -m venv product-recommendation-system-env
source product-recommendation-system-env/bin/activate  # On Windows: .\product-recommendation-system-env\Scripts\activate
pip install -r requirements.txt