iBrokeTheCode/Multimodal_Product_Classification
0
1import gradio as gr2 3from app_predictor import predict4 5# 📌 CUSTOM CSS6css_code = """7#footer-container {8 position: fixed;9 bottom: 0;10 left: 0;11 right: 0;12 z-index: 1000;13 background-color: var(--background-fill-primary);14 padding: var(--spacing-md);15 border-top: 1px solid var(--border-color-primary);16 text-align: center;17}18 19.gradio-container {20 padding-bottom: 70px !important;21}22 23.center {24 text-align: center;25}26"""27 28 29def update_inputs(mode: str):30 if mode == "Multimodal":31 return gr.Textbox(visible=True), gr.Image(visible=True)32 elif mode == "Text Only":33 return gr.Textbox(visible=True), gr.Image(visible=False)34 elif mode == "Image Only":35 return gr.Textbox(visible=False), gr.Image(visible=True)36 else: # Default case37 return gr.Textbox(visible=True), gr.Image(visible=True)38 39 40# 📌 USER INTERFACE41with gr.Blocks(42 title="Multimodal Product Classification",43 theme=gr.themes.Ocean(),44 css=css_code,45) as demo:46 with gr.Tabs():47 # 📌 APP TAB48 with gr.TabItem("🚀 App"):49 with gr.Row(elem_classes="center"):50 gr.HTML("""51 <div>52 <h1>🛍️ Multimodal Product Classification</h1>53 </div>54 <br><br>55 """)56 57 with gr.Row(equal_height=True):58 # 📌 CLASSIFICATION INPUTS COLUMN59 with gr.Column():60 with gr.Column():61 gr.Markdown("## 📝 Classification Inputs")62 63 mode_radio = gr.Radio(64 choices=["Multimodal", "Image Only", "Text Only"],65 value="Multimodal",66 label="Choose Classification Mode:",67 )68 69 text_input = gr.Textbox(70 label="Product Description:",71 placeholder="e.g., Apple iPhone 15 Pro Max 256GB",72 lines=1,73 )74 75 image_input = gr.Image(76 label="Product Image",77 type="filepath",78 visible=True,79 height=300,80 width="100%",81 )82 83 classify_button = gr.Button(84 "✨ Classify Product", variant="primary"85 )86 87 # 📌 RESULTS COLUMN88 with gr.Column():89 with gr.Column():90 gr.Markdown("## 📊 Results")91 92 gr.Markdown(93 """**💡 How to use this app**94 95 This app classifies a product based on its description and image.96 - **Multimodal:** The most accurate mode, using both the image and a detailed description for prediction.97 - **Image Only:** Highly effective for visual products, relying solely on the product image.98 - **Text Only:** Less precise, this mode requires a very descriptive and specific product description to achieve good results.99 """100 )101 102 gr.HTML("<hr>")103 104 output_label = gr.Label(105 label="Predict category", num_top_classes=5106 )107 108 # 📌 EXAMPLES SECTION109 gr.Examples(110 examples=[111 [112 "Multimodal",113 'Laptop Asus - 15.6" / CPU I9 / 2Tb SSD / 32Gb RAM / RTX 2080',114 "./assets/sample2.jpg",115 ],116 [117 "Multimodal",118 "Red Electric Guitar – Stratocaster Style, 6-String, White Pickguard, Solid-Body, Ideal for Rock & Roll",119 "./assets/sample1.jpg",120 ],121 [122 "Multimodal",123 "Portable Wireless Speaker / JBL / Black / High Quality Sound",124 "./assets/sample3.jpg",125 ],126 ],127 label="Select an example to pre-fill the inputs, then click the 'Classify Product' button.",128 inputs=[mode_radio, text_input, image_input],129 # outputs=output_label,130 # fn=predict,131 # cache_examples=True,132 )133 134 # 📌 ABOUT TAB135 with gr.TabItem("ℹ️ About"):136 gr.Markdown("""137## Project Overview138 139- This project is a multimodal product classification system for Best Buy products. 140- The core objective is to categorize products using both their text descriptions and images. 141- The system was trained on a dataset of **almost 50,000** products and their corresponding images to generate embeddings and train the classification models.142 143<br>144 145## Technical Workflow146 1471. **Data Preprocessing:** Product descriptions and images are extracted from the dataset, and a `categories.json` file is used to map product IDs to human-readable category names.1482. **Embedding Generation:**149 - **Text:** A pre-trained `SentenceTransformer` model (`all-MiniLM-L6-v2`) is used to generate dense vector embeddings from the product descriptions.150 - **Image:** A pre-trained computer vision model from the Hugging Face `transformers` library (`TFConvNextV2Model`) is used to extract image features.1513. **Model Training:** The generated text and image embeddings are then used to train a multi-layer perceptron (MLP) model for classification. Separate models were trained for text-only, image-only, and multimodal (combined embeddings) classification.1524. **Deployment:** The trained models are deployed via a Gradio web interface, allowing for live prediction on new product data.153 154<br>155 156> **💡 Want to explore the process in detail?** 157> See the full 👉 [Jupyter notebook](https://huggingface.co/spaces/iBrokeTheCode/Multimodal_Product_Classification/blob/main/notebook_guide.ipynb) 👈️ for an end-to-end walkthrough, including Exploratory Data Analysis, embeddings generation, models training, evaluation, and model selection.158""")159 160 # 📌 MODEL TAB161 with gr.TabItem("🎯 Model"):162 gr.Markdown("""163## Model Details164The final classification is performed by a Multi-layer Perceptron (MLP) trained on the embeddings. This architecture allows the model to learn the relationships between the textual and visual features.165 166<br>167 168## Performance Summary169 170The following table summarizes the performance of all models trained in this project.171 172<br>173 174| Model | Modality | Accuracy | Macro Avg F1-Score | Weighted Avg F1-Score |175| :------------------ | :----------- | :------- | :----------------- | :-------------------- |176| Random Forest | Text | 0.90 | 0.83 | 0.90 |177| Logistic Regression | Text | 0.90 | 0.84 | 0.90 |178| Random Forest | Image | 0.80 | 0.70 | 0.79 |179| Random Forest | Combined | 0.89 | 0.79 | 0.89 |180| Logistic Regression | Combined | 0.89 | 0.83 | 0.89 |181| **MLP** | **Image** | **0.84** | **0.77** | **0.84** |182| **MLP** | **Text** | **0.92** | **0.87** | **0.92** |183| **MLP** | **Combined** | **0.92** | **0.85** | **0.92** |184 185<br>186 187## Conclusion188 189- Based on the overall results, the MLP models consistently outperformed their classical machine learning counterparts, demonstrating their ability to learn intricate, non-linear relationships within the data.190- Both the Text MLP and Combined MLP models achieved the highest accuracy and weighted F1-score, confirming their superior ability to classify the products.191- This modular approach demonstrates the ability to handle various data modalities and evaluate the contribution of each to the final prediction.192""")193 194 # 📌 FOOTER195 # gr.HTML("<hr>")196 with gr.Row(elem_id="footer-container"):197 gr.HTML("""198<div>199 <b>Connect with me:</b> 💼 <a href="https://www.linkedin.com/in/alex-turpo/" target="_blank">LinkedIn</a> • 200 🐱 <a href="https://github.com/iBrokeTheCode" target="_blank">GitHub</a> • 201 🤗 <a href="https://huggingface.co/iBrokeTheCode" target="_blank">Hugging Face</a>202 </div>203""")204 205 # 📌 EVENT LISTENERS206 mode_radio.change(207 fn=update_inputs,208 inputs=mode_radio,209 outputs=[text_input, image_input],210 )211 212 classify_button.click(213 fn=predict, inputs=[mode_radio, text_input, image_input], outputs=output_label214 )215 216 217demo.launch()218