DanielLevenstein/Helmet_CNN_Data_Quality_Case_Study

Project: Model Cross-Data Validation Purpose

• —Compare the performance of four helmet-detection CNNs trained on different datasets using holdout data from the second two datasets for testing.
• —Demonstrate how dataset quality and preprocessing choices affect final model performance.

Datasets

Training notebooks: Helmet_CNN_Model_Training_Notebooks | Dataset | Source | Datasets Type | Samples Used | Model Accuracy | | -------- | ------------------------------------------------------------------------------------------------------------------------------- | -------------- | ------------ | -------------- | | Dataset0 | Great Learning:HelmNet_ImageProcessing_Notebook | Classification | ~600 | ~76% | | Dataset1 | Kaggle:on-vehicle-helmet-detection-dataset | Classification | ~500 | ~70% | | Dataset2 | Kaggle:helmet-detection | Bounding Boxes | ~250 | ~93% | | Dataset3 | Kaggle:hard-hat-detection | Bounding Boxes | ~3000 | ~93% |

Streamlit App

This streamlit app was inspired by a model I built for a school project at UT that performed really well on training data but poorly on real-world data. Datasets 1, 2 and 3 were all attempts to improve the performance of this initial model. I have since moved the cross-model validation out of my streamlit app and into a standalone validation notebook which is uploaded to GitHub. Model Cross Validation Notebook

Preprocessing

• —Classification datasets (Dataset0, Dataset1)
• —All images resized to 200×200 pixels prior to training.
• —Detection datasets (Dataset2, Dataset3)
• —Each bounding box is processed to create training crops.
• —For each object instance, a 100×100 pixel image was saved with the object's center positioned as close as possible to the crop center (given original bounding-box location).
• —Holdout
• —A holdout set was extracted from Dataset2 and Dataset3 and saved prior to any training; used only for cross-model validation.
• —The final validation set consisted of 50 images taken from each of these holdout sets.
• —Due to the small sample size of Dataset2, it was not possible to create a larger final validation set.

Validation

• —All four models are evaluated based on from a hold-out dataset from Dataset2, and Dataset3
• —Because Dataset2 is significantly smaller than Dataset3, a sample of 50 images was taken from each holdout set for evaluation.
• —When evaluating Model0 and Model1, the validation data is resized to 200x200 prior to evaluation.

Misclassified Images

• —Dataset2 misclassified images involving individuals wearing headcoverings as containing helmets when they did not.
• —Dataset3 misclassified images include images which included helmets in the background.

Cross-Validation Results

Conclusion

The performance of Model2 and Model3 are close, with Model2 performing slightly better on the final validation set.

Model3 was a significantly bigger dataset but contained a lot of inverted or mirrored images. This might have contributed to the worse performance. Also, most of the no-helmet samples in Dataset3 were managers, possibly leading the model to attribute nicer clothing to that class. Given the small size of the validation set, it's important not to read too much into these results.

After the validation data was normalized, the performance of Model0 improved enough to surpass Model1 on all metrics. This was surprising given how limited the no-helmet class was in this dataset. Model1 consisted of only two individuals and was taken by hand, but on initial inspection still felt like a higher-quality dataset than Dataset0.

Dataset origins and preprocessing steps played crucial roles in final model performance. Dataset0 originated from an internal UT project, whereas Datasets1 through Dataset3 were publicly available Kaggle datasets. Consistent resizing and cropping across datasets were key to reducing input-size variation and isolating dataset-quality effects.

It's worth noting that a difference in performance of less than 2% percentage points on the small validation dataset should be disregarded for conclusive claims. In addition, standardization of image size and removal of low quality features had an effect on the final results.