rotemknat/israeli_supermarkets_pricing

🛒 Smart Shopping: Israeli Supermarket Price Predictor

Welcome! This project is an end-to-end Machine Learning exploration of the Israeli Supermarkets 2024 dataset. We’ve built a robust pipeline that doesn't just look at prices, but understands the patterns behind them—predicting exact costs and categorizing items into "Low," "Medium," and "High" price tiers.

📺 Project Walkthrough

Want to see the logic in action? Check out the video presentation below:

<video src="https://huggingface.co/datasets/rotemknat/israelisupermarketspricing/resolve/main/Rotem%20Assignment%20%232%20-%20Classification%2C%20Regression%2C%20Clustering%2C%20Evaluation%20-%20Colab%20-%20Google%20Chrome%20-%201%20May%202026.mp4" controls="controls" style="max-width: 720px; border-radius: 12px; border: 1px solid #ddd;"></video>

📊 The Data Story

• —Source: Israeli Supermarkets 2024 (Kaggle)
• —The Scale: We started with a massive pool of 10M+ rows. Through careful selection and cleaning, we focused on a high-quality sample of ~39,000 rows to keep our models sharp and efficient.
• —Core Features: We analyzed everything from itemcode and manufacturer to unitofmeasure and temporal data.
• —The Mission: Master the art of predicting itemprice (Regression) and price-tier categorization (Classification).

🛠️ Exploratory Data Analysis (EDA)

Before the "learning" starts, we had to clean up the noise. Our EDA focused on:

• —Filtering Noise: Dropped columns with >90% missing data and removed 10k+ duplicate entries.
• —Outlier Defense: We used the IQR method to ensure our model wasn't distracted by extreme price anomalies.
• —Sanity Checks: Removed impossible data points, like items with zero or negative quantities.
• —Refining the Target: Capped itemprice at the 99th percentile to create a more stable learning target.

.png)

🧪 Feature Engineering

We turned raw numbers into meaningful insights:

• —📅 The "When": Extracted specific time features (Day of week, Month, etc.) to see if prices fluctuate on weekends or holidays.
• —🏷️ The "Who": Applied Frequency Encoding to handle high-variety fields like item names and manufacturers.
• —🤖 The "Groups": We used K-Means Clustering to let the data group itself. These "hidden" clusters became a powerful new feature for our models.

📈 Regression Models

We progressed from a simple baseline to high-performance ensemble models.

Baseline Model: Linear Regression

• —Features: unitofmeasureprice, quantity, allowdiscount
• —Metrics: MAE: (Value) | RMSE: (Value) | R2: (Value)

.png)

The Contenders (Improved Models)

1. 1.Linear Regression (Improved): Leveraged our new engineered features.

1. 1.Random Forest Regressor: Captured complex, non-linear relationships.

1. 1.Gradient Boosting Regressor: Our top performer!

🏆 Regression Winner: Random Forest Regressor

Reasoning: It demonstrated the highest R2 score and lowest error metrics, proving that sequential learning is incredibly effective for retail price prediction.

🎯 Classification Models

We split the items into three balanced buckets: Low, Medium, and High using quantile binning.

The Classifiers

1. 1.Logistic Regression: Our statistical baseline.

1. 1.Random Forest Classifier: A robust ensemble approach.

1. 1.Gradient Boosting Classifier: A strong competitor in precision.

🏆 Classification Winner: Random Forest Classifier

Reasoning: (Explain why this model performed best, referencing precision, recall, and F1-score).

💡 Key Considerations: Precision vs. Recall

In retail analytics, the "cost" of a mistake varies:

• —Precision: If we label an item "High Price," how sure are we? High precision avoids over-pricing errors.
• —Recall: Are we catching all the "High Price" items? High recall ensures no premium goods are missed.
• —The Trade-off: (Add your specific critical error type here.)

🏁 Conclusion & Lessons Learned

• —Feature Power: Data context (like K-Means clusters) often provides a bigger boost than just "tuning the knobs."
• —Handling Scale: Processing millions of rows requires a balance of memory management and smart sampling.
• —Iteration is Key: The path from a basic Linear Regression to a Gradient Boosting winner highlights the importance of the iterative Data Science lifecycle.

🧑‍💻 How to Use

To use the winning models from this project, you can download the .pkl files and load them using the following code:

import joblib
import pandas as pd

# 1. Load the Regression Winner (Random Forest)
regressor = joblib.load('random_forest_regressor_model.pkl')

# 2. Load the Classification Winner (Gradient Boosting)
classifier = joblib.load('gradient_boosting_classifier_model.pkl')