sunkaiwen/places-water-resnet18-automl

places-water-resnet18-automl

Binary scene classifier — does this image contain water? A pretrained resnet34 fine-tuned with AutoGluon MultiModalPredictor, selected by a budgeted search over how much of the backbone to unfreeze (optim.peft) together with the backbone depth and the optimiser hyperparameters.

Built for Homework 2 of 24-679 Designing with AI.

1. Purpose

Given a 224×224 RGB photograph of a place, predict water or no_water.

Intended use: coursework and demonstration of transfer learning with a budgeted neural-architecture search on a small image dataset. Out of scope: flood detection, safety systems, environmental monitoring, or any decision with a consequence. The evidence base is roughly two dozen photographs.

2. Data origin and splits

Source: `ssg1/places-water-binary`, collected by a classmate for HW1. I am not the dataset author and did not contribute to its collection.

The shipped validation split holds 5 images. Five images cannot rank 8 configurations — one image moves the score 20 points — so selection used an internal split carved out of train and grouped by parent_id, and the shipped validation images were folded into the final test set instead. Grouping matters because the training split is flips and crops of roughly 23 originals; an ungrouped split would let a flipped copy of a training image select the model.

3. Method

The Week 4 lecture fine-tuned a ResNet-18 inside AutoGluon and made the amount of fine-tuning explicit through optim.peft. This model follows that method directly:

Selected: `peft = None` on a resnet34 backbone.

4. Search

The first four configurations cover all four peft settings exactly once, so the comparison below is guaranteed rather than left to the random draws.

Selected configuration:

{
  "peft": "None",
  "backbone": "resnet34",
  "lr": 0.002218163430073746,
  "lr_decay": 1.0,
  "batch_size": 32
}

What the `peft` setting was worth:

search_results.csv in this repo records every configuration and its score.

5. Metrics

The test set is eleven images. Its confidence interval is roughly 25 percentage points wide, so it confirms the model is not broken and nothing more. The internal-validation figure, computed on 102 images, is the number to use when comparing configurations.

6. Limitations

1. 1.~23 original photographs. Everything else is augmentation. The grouped split accounts for this; the raw image count does not.
2. 2.Eleven test images. Any comparison against another model on this test set is noise.
3. 3."Water" is one annotator's judgement. A puddle, a fountain, a wet street — the boundary was drawn by the dataset author alone, with no second opinion.
4. 4.Scene bias. The images come from one photo collection with its own framing, lighting and geography. Performance on aerial imagery, underwater shots, or night scenes is unknown and probably poor.
5. 5.ImageNet inheritance. The backbone carries whatever biases ImageNet pretraining carries; a frozen-filter fine-tune inherits them almost entirely.
6. 6.Possible shortcut. If the EDA colour panel showed clean separation, part of this score may come from mean blue level rather than from recognising water.

7. Ethical considerations

The images are of places, not identifiable people. The realistic harm is over-trust: a binary water detector trained on two dozen photographs will fail on anything outside its narrow visual domain, and should never be placed anywhere a false negative matters.

8. License

CC-BY-4.0, matching the source dataset's terms. Attribution to the dataset author for the underlying images. The resnet34 backbone is ImageNet-pretrained via timm.

9. Compute budget

Tesla T4 · 8 configurations in 3.3 minutes, plus a 42-second retrain of the winner.

10. AI usage disclosure

Generative AI (Claude, Anthropic) was used as a coding and writing assistant: it drafted the search harness, the AutoGluon configuration, the plotting code and the prose of this card. The method — fine-tuning a pretrained ResNet rather than training from scratch, searching peft as the architectural axis, and grouping the validation split by parent_id — was reviewed and accepted by me, and every cell was executed and checked before upload. I am responsible for the content.

11. How to use

import pandas as pd
from huggingface_hub import snapshot_download
from autogluon.multimodal import MultiModalPredictor

predictor = MultiModalPredictor.load(snapshot_download("sunkaiwen/places-water-resnet18-automl") + "/predictor")
print(predictor.predict(pd.DataFrame({"image": ["my_photo.jpg"]})))