strickvl/redaction-detector

1I've been working through the first two lessons of2[the fastai course](https://course.fast.ai/). For lesson one I trained a model3to recognise my cat, Mr Blupus. For lesson two the emphasis is on getting those4models out in the world as some kind of demo or application.5[Gradio](https://gradio.app) and6[Huggingface Spaces](https://huggingface.co/spaces) makes it super easy to get a7prototype of your model on the internet.8 9This MVP app runs two models to mimic the experience of what a final deployed10version of the project might look like.11 12- The first model (a classification model trained with fastai, available on the13  Huggingface Hub14  [here](https://huggingface.co/strickvl/redaction-classifier-fastai) and15  testable as a standalone demo16  [here](https://huggingface.co/spaces/strickvl/fastai_redaction_classifier)),17  classifies and determines which pages of the PDF are redacted. I've written18  about how I trained this model [here](https://mlops.systems/fastai/redactionmodel/computervision/datalabelling/2021/09/06/redaction-classification-chapter-2.html).19- The second model (an object detection model trained using [IceVision](https://airctic.com/) (itself20  built partly on top of fastai)) detects which parts of the image are redacted.21  This is a model I've been working on for a while and I described my process in22  a series of blog posts (see below).23 24This MVP app does several things:25 26- it extracts any pages it considers to contain redactions and displays that27  subset as an [image carousel](https://gradio.app/docs/#o_carousel). It also28  displays some text alerting you to which specific pages were redacted.29- if you click the "Analyse and extract redacted images" checkbox, it will:30  - pass the pages it considered redacted through the object detection model31  - calculate what proportion of the total area of the image was redacted as32    well as what proportion of the actual content (i.e. excluding margins etc33    where there is no content)34  - create a PDF that you can download that contains only the redacted images,35    with an overlay of the redactions that it was able to identify along with36    the confidence score for each item.37 38## The Dataset39 40I downloaded a few thousand publicly-available FOIA documents from a government41website. I split the PDFs up into individual `.jpg` files and then used42[Prodigy](https://prodi.gy/) to annotate the data. (This process was described43in44[a blogpost written last45year](https://mlops.systems/fastai/redactionmodel/computervision/datalabelling/2021/09/06/redaction-classification-chapter-2.html).)46For the object detection model, the process was quite a bit more involved and I47direct you to the series of articles referenced below in the 'Further Reading' section.48 49## Training the model50 51I trained the classification model with fastai's flexible `vision_learner`, fine-tuning52`resnet18` which was both smaller than `resnet34` (no surprises there) and less53liable to early overfitting. I trained the model for 10 epochs.54 55The object detection model is trained using IceVision, with VFNet as the56model and `resnet50` as the backbone. I trained the model for 50 epochs and57reached 89% accuracy on the validation data.58 59## Further Reading60 61This initial dataset spurred an ongoing interest in the domain and I've since62been working on the problem of object detection, i.e. identifying exactly which63parts of the image contain redactions.64 65Some of the key blogs I've written about this project:66 67- How to annotate data for an object detection problem with Prodigy68  ([link](https://mlops.systems/redactionmodel/computervision/datalabelling/2021/11/29/prodigy-object-detection-training.html))69- How to create synthetic images to supplement a small dataset70  ([link](https://mlops.systems/redactionmodel/computervision/python/tools/2022/02/10/synthetic-image-data.html))71- How to use error analysis and visual tools like FiftyOne to improve model72  performance73  ([link](https://mlops.systems/redactionmodel/computervision/tools/debugging/jupyter/2022/03/12/fiftyone-computervision.html))74- Creating more synthetic data focused on the tasks my model finds hard75  ([link](https://mlops.systems/tools/redactionmodel/computervision/2022/04/06/synthetic-data-results.html))76- Data validation for object detection / computer vision (a three part series —77  [part 1](https://mlops.systems/tools/redactionmodel/computervision/datavalidation/2022/04/19/data-validation-great-expectations-part-1.html),78  [part 2](https://mlops.systems/tools/redactionmodel/computervision/datavalidation/2022/04/26/data-validation-great-expectations-part-2.html),79  [part 3](https://mlops.systems/tools/redactionmodel/computervision/datavalidation/2022/04/28/data-validation-great-expectations-part-3.html))80