SparshSG/Dish-Decoder

1# -*- coding: utf-8 -*-2"""model_training.ipynb3 4Automatically generated by Colaboratory.5 6Original file is located at7    https://colab.research.google.com/drive/1LgqvdLV1teCsAi6qjR_BBVt4TwX7vx9J8 9<a href="https://colab.research.google.com/github/gauravreddy08/food-vision/blob/main/model_training.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>10 11# **Food Vision** 🍔12 13As an introductory project to myself, I built an **end-to-end CNN Image Classification Model** which identifies the food in your image.14 15I worked out with a pretrained Image Classification Model that comes with Keras and then retrained it on the infamous **Food101 Dataset**.16 17 18**Fun Fact :**19 20The Model actually beats the DeepFood Paper's model which also trained on the same dataset.21 22The Accuracy of [**DeepFood**](https://arxiv.org/abs/1606.05675) was **77.4%** and our model's is **85%**. Difference of **8%** ain't much but the interesting thing is, DeepFood's model took 2-3 days to train while our's was around 60min.23 24> **Dataset :** `Food101`25 26> **Model :** `EfficientNetB1`27 28## **Setting up the Workspace**29 30* Checking the GPU31* Mounting Google Drive32* Importing Tensorflow33* Importing other required Packages34 35### **Checking the GPU**36 37For this Project we will working with **Mixed Precision**. And mixed precision works best with a with a GPU with compatibility capacity **7.0+**.38 39At the time of writing, colab offers the following GPU's :40* Nvidia K8041* **Nvidia T4**42* Nvidia P10043 44Colab allocates a random GPU everytime we factory reset runtime. So you can reset the runtime till you get a **Tesla T4 GPU** as T4 GPU has a rating 7.5.45 46> In case using local hardware, use a GPU with rating 7.0+ for better results.47 48Run the below cell to see which GPU is allocated to you.49"""50 51!nvidia-smi -L52 53"""54### **Mounting Google Drive**55 56 57"""58 59from google.colab import drive60drive.mount('/content/drive')61 62"""### **Importing Tensorflow**63 64At the time of writing, `tesnorflow 2.5.0` has a bug with EfficientNet Models. [Click Here](https://github.com/tensorflow/tensorflow/issues/49725) to get more info about the bug. Hopefully tensorflow fixes it soon.65 66So the below code is used to downgrade the version to `tensorflow 2.4.1`, it will take a moment to uninstall the previous version and install our required version.67 68> You need to restart the **Runtime** after required version of tensorflow is installed.69 70**Note :** Restarting runtime won't assign you a new GPU.71"""72 73#!pip install tensorflow==2.4.174import tensorflow as tf75print(tf.__version__)76 77"""### **Importing other required Packages**"""78 79import pandas as pd80import numpy as np81import matplotlib.pyplot as plt82import datetime83import os84import tensorflow_datasets as tfds85import seaborn as sn86 87"""#### **Importing `helper_fuctions`**88 89The `helper_functions.py` is a python script created by me. Which has some important functions I use frequently while building Deep Learning Models.90"""91 92!wget https://raw.githubusercontent.com/sg-sparsh-goyal/extras/main/helper_function.py93 94from helper_function import plot_loss_curves, load_and_prep_image95 96"""## **Getting the Data Ready**97 98The Dataset used is **Food101**, which is available on both Kaggle and Tensorflow.99 100In the below cells we will be importing Datasets from `Tensorflow Datasets` Module.101 102"""103 104# Prints list of Datasets avaible in Tensorflow Datasets Module105 106dataset_list = tfds.list_builders()107dataset_list[:10]108 109"""### **Importing Food101 Dataset**110 111**Disclaimer :**112The below cell will take time to run, as it will be downloading113**4.65GB data** from **Tensorflow Datasets Module**.114 115So do check if you have enough **Disk Space** and **Bandwidth Cap** to run the below cell.116"""117 118(train_data, test_data), ds_info = tfds.load(name='food101',119                                             split=['train', 'validation'],120                                             shuffle_files=False,121                                             as_supervised=True,122                                             with_info=True)123 124"""## **Becoming One with the Data**125 126One of the most important steps in building any ML or DL Model is to **become one with the data**.127 128Once you get the gist of what type of data your dealing with and how it is structured, everything else will fall in place.129"""130 131ds_info.features132 133class_names = ds_info.features['label'].names134class_names[:10]135 136train_one_sample = train_data.take(1)137 138train_one_sample139 140for image, label in train_one_sample:141  print(f"""142  Image Shape : {image.shape}143  Image Datatype : {image.dtype}144  Class : {class_names[label.numpy()]}145  """)146 147image[:2]148 149tf.reduce_min(image), tf.reduce_max(image)150 151plt.imshow(image)152plt.title(class_names[label.numpy()])153plt.axis(False);154 155"""## **Preprocessing the Data**156 157Since we've downloaded the data from TensorFlow Datasets, there are a couple of preprocessing steps we have to take before it's ready to model.158 159More specifically, our data is currently:160 161* In `uint8` data type162* Comprised of all differnet sized tensors (different sized images)163* Not scaled (the pixel values are between 0 & 255)164 165Whereas, models like data to be:166 167* In `float32` data type168* Have all of the same size tensors (batches require all tensors have the same shape, e.g. `(224, 224, 3)`)169* Scaled (values between 0 & 1), also called normalized170 171To take care of these, we'll create a `preprocess_img()` function which:172 173* Resizes an input image tensor to a specified size using [`tf.image.resize()`](https://www.tensorflow.org/api_docs/python/tf/image/resize)174* Converts an input image tensor's current datatype to `tf.float32` using [`tf.cast()`](https://www.tensorflow.org/api_docs/python/tf/cast)175"""176 177def preprocess_img(image, label, img_size=224):178  image = tf.image.resize(image, [img_size, img_size])179  image = tf.cast(image, tf.float16)180  return image, label181 182# Trying the preprocess function on a single image183 184preprocessed_img = preprocess_img(image, label)[0]185preprocessed_img186 187train_data = train_data.map(preprocess_img, tf.data.AUTOTUNE)188train_data = train_data.shuffle(buffer_size=1000).batch(32).prefetch(tf.data.AUTOTUNE)189 190test_data = test_data.map(preprocess_img, tf.data.AUTOTUNE)191test_data = test_data.batch(32)192 193train_data194 195test_data196 197"""## **Building the Model : EfficientNetB1**198 199 200### **Getting the Callbacks ready**201As we are dealing with a complex Neural Network (EfficientNetB0) its a good practice to have few call backs set up. Few callbacks I will be using throughtout this Notebook are :202 * **TensorBoard Callback :** TensorBoard provides the visualization and tooling needed for machine learning experimentation203 204 * **EarlyStoppingCallback :** Used to stop training when a monitored metric has stopped improving.205 206 * **ReduceLROnPlateau :** Reduce learning rate when a metric has stopped improving.207 208 209 We already have **TensorBoardCallBack** function setup in out helper function, all we have to do is get other callbacks ready.210"""211 212from helper_function import create_tensorboard_callback213 214# EarlyStopping Callback215 216early_stopping_callback = tf.keras.callbacks.EarlyStopping(restore_best_weights=True, patience=3, verbose=1, monitor="val_accuracy")217 218# ReduceLROnPlateau Callback219 220lower_lr = tf.keras.callbacks.ReduceLROnPlateau(factor=0.2,221                                                monitor='val_accuracy',222                                                min_lr=1e-7,223                                                patience=0,224                                                verbose=1)225 226"""227 228### **Mixed Precision Training**229Mixed precision is used for training neural networks, reducing training time and memory requirements without affecting the model performance.230 231More Specifically, in **Mixed Precision** we will setting global dtype as `mixed_float16`. Because modern accelerators can run operations faster in the 16-bit dtypes, as they have specialized hardware to run 16-bit computations and 16-bit dtypes can be read from memory faster.232 233To know more about Mixed Precision, [**click here**](https://www.tensorflow.org/guide/mixed_precision)"""234 235from tensorflow.keras import mixed_precision236mixed_precision.set_global_policy(policy='mixed_float16')237 238mixed_precision.global_policy()239 240"""241 242### **Building the Model**"""243 244from tensorflow.keras import layers245from tensorflow.keras.layers.experimental import preprocessing246 247# Create base model248input_shape = (224, 224, 3)249base_model = tf.keras.applications.EfficientNetB1(include_top=False)250 251# Input and Data Augmentation252inputs = layers.Input(shape=input_shape, name="input_layer")253x = base_model(inputs)254 255x = layers.GlobalAveragePooling2D(name="pooling_layer")(x)256x = layers.Dropout(.3)(x)257 258x = layers.Dense(len(class_names))(x)259outputs = layers.Activation("softmax")(x)260model = tf.keras.Model(inputs, outputs)261 262# Compiling the model263model.compile(loss="sparse_categorical_crossentropy",264              optimizer=tf.keras.optimizers.Adam(0.001),265              metrics=["accuracy"])266 267model.summary()268 269history = model.fit(train_data,270                    epochs=50,271                    steps_per_epoch=len(train_data),272                    validation_data=test_data,273                    validation_steps=int(0.15 * len(test_data)),274                    callbacks=[create_tensorboard_callback("training-logs", "EfficientNetB1-"),275                               early_stopping_callback,276                               lower_lr])277 278# Saving the model279model.save("/content/drive/My Drive/FinalModel.hdf5")280 281# Saving the model282model.save("FoodVision.hdf5")283 284plot_loss_curves(history)285 286model.evaluate(test_data)287 288"""## **Evaluating our Model**"""289 290# Commented out IPython magic to ensure Python compatibility.291# %load_ext tensorboard292# %tensorboard --logdir training-logs293 294pred_probs = model.predict(test_data, verbose=1)295len(pred_probs), pred_probs.shape296 297pred_classes = pred_probs.argmax(axis=1)298pred_classes[:10], len(pred_classes), pred_classes.shape299 300# Getting true labels for the test_data301 302y_labels = []303test_images = []304for images, labels in test_data.unbatch():305  y_labels.append(labels.numpy())306y_labels[:10]307 308# Predicted Labels vs. True Labels309pred_classes==y_labels310 311"""### **Sklearn's Accuracy Score**"""312 313from sklearn.metrics import accuracy_score314 315sklearn_acc = accuracy_score(y_labels, pred_classes)316sklearn_acc317 318"""### **Confusion Matrix**319A confusion matrix is a table that is often used to describe the performance of a classification model (or "classifier") on a set of test data for which the true values are known320"""321 322cm = tf.math.confusion_matrix(y_labels, pred_classes)323 324plt.figure(figsize = (100, 100));325sn.heatmap(cm, annot=True,326           fmt='',327           cmap='Purples');328 329"""### **Model's Class-wise Accuracy Score**"""330 331from sklearn.metrics import classification_report332report = (classification_report(y_labels, pred_classes, output_dict=True))333 334# Create empty dictionary335class_f1_scores = {}336# Loop through classification report items337for k, v in report.items():338  if k == "accuracy": # stop once we get to accuracy key339    break340  else:341    # Append class names and f1-scores to new dictionary342    class_f1_scores[class_names[int(k)]] = v["f1-score"]343class_f1_scores344 345report_df = pd.DataFrame(class_f1_scores, index = ['f1-scores']).T346 347report_df = report_df.sort_values("f1-scores", ascending=True)348 349import matplotlib.pyplot as plt350 351fig, ax = plt.subplots(figsize=(12, 25))352scores = ax.barh(range(len(report_df)), report_df["f1-scores"].values)353ax.set_yticks(range(len(report_df)))354plt.axvline(x=0.85, linestyle='--', color='r')355ax.set_yticklabels(class_names)356ax.set_xlabel("f1-score")357ax.set_title("F1-Scores for 10 Different Classes")358ax.invert_yaxis(); # reverse the order359 360"""### **Predicting on our own Custom images**361 362Once we have our model ready, its cruicial to evaluate it on our custom data : the data our model has never seen.363 364Training and evaluating a model on train and test data is cool, but making predictions on our own realtime images is another level.365 366 367"""368 369import os370 371directory_path = "/content/drive/MyDrive/FoodVisionModels/Custom Images"372os.makedirs(directory_path, exist_ok=True)373 374custom_food_images = [directory_path + img_path for img_path in os.listdir(directory_path)]375custom_food_images376 377import os378import matplotlib.pyplot as plt379 380def pred_plot_custom(folder_path):381    custom_food_images = [folder_path + img_path for img_path in os.listdir(folder_path) if os.path.isfile(os.path.join(folder_path, img_path))]382 383    for img in custom_food_images:384        img = load_and_prep_image(img, scale=False)385        pred_prob = model.predict(tf.expand_dims(img, axis=0))386        pred_class = class_names[pred_prob.argmax()]387        top_5_i = (pred_prob.argsort())[0][-5:][::-1]388        values = pred_prob[0][top_5_i]389        labels = []390 391        for x in range(5):392            labels.append(class_names[top_5_i[x]])393 394        fig, ax = plt.subplots(1, 2, figsize=(15, 5))395 396        # Plotting Image397        ax[0].imshow(img/255.)398        ax[0].set_title(f"Prediction: {pred_class}   Probability: {pred_prob.max():.2f}")399        ax[0].axis('off')400 401        # Plotting Models Top 5 Predictions402        ax[1].bar(labels, values, color='orange')403        ax[1].set_title('Top 5 Predictions')404 405        plt.show()406 407pred_plot_custom("/content/drive/MyDrive/FoodVisionModels/Custom Images/")408 409