hansaka01/crophelth_finetune_2nd_stage

CropHelth Round-2 Fine-Tune Dataset

Author: Hansaka Rasanjana

Project Context: First-year IoT group mini-project at the Sri Lanka Institute of Information Technology (SLIIT)

5,457 brand-new leaf images across 60 of the 112 crop-disease classes — zero overlap with the original 134,016-image crophelth training set. Put together to fine-tune our round-1 model (train_report.json: accuracy 0.9484, macro F1 0.8889), with per-class sampling weighted heavily toward our weak classes.

Why we're fine-tuning on new images

Round 2 uses only images the model has never seen before:

• —New public datasets (different farms, countries, camera conditions) + a small batch of verified web images for our weakest classes.
• —Exact-MD5 dedupe run across the entire set (184 duplicates dropped).
• —The 112-class label space stays identical: classes without new data simply aren't in round 2, so their learned weights are safely preserved (in Keras, class heads with no samples in a batch receive zero gradient).

Layout

images/<class_code>/NNNNNN.jpg      5,457 images, 60 classes
finetune_index.csv                  file,label,class_index,split   (stratified 90/10, seed 42)
class_weights.json                  Keras class_weight dict, derived from round-1 F1
                                    w = clamp(1/(0.2+f1), 0.5, 3.0)  -> weak classes weigh more
summary.json                        per-class kept/cap/round1-f1 + missing-class list

How classes were balanced (per class, from round-1 F1)

New sources (all CC-BY / CC-BY-SA, properly attributed)

Classes without new public data (52) — staying at round-1 weights

Couldn't find clean, honest new public sources for these across Mendeley, Zenodo, GitHub, or general web searches: mostly lemon diseases (15), tea pests/blights (5), banana diseases and nutrient deficiencies (9: black sigatoka, yb sigatoka, fusarium wilt, bract mosaic, insect pest, boron/calcium/iron/manganese deficiencies), plus single-class crops like blueberry, cherry, peach, raspberry, soybean, squash, strawberry, and a couple of miscellaneous classes.

Their softmax heads are frozen in round 2 (no samples mean no gradient), so nothing gets lost—they remain completely functional. We'll need to grab our own field photos for these moving forward.

Usage (Keras)

import pandas as pd, json, tensorflow as tf
df = pd.read_csv("finetune_index.csv")
weights = json.load(open("class_weights.json"))
# build a tf.data pipeline from df[df.split=="train"] (same pattern as round 1),
# then:
model.compile(optimizer=tf.keras.optimizers.Adam(1e-5),
              loss=tf.keras.losses.SparseCategoricalCrossentropy(),
              metrics=["accuracy"])
model.fit(train_ds, validation_data=val_ds, epochs=3-5, class_weight=weights)

Recommended setup for round 2: Stick with the same EfficientNetB0/B2 architecture, drop down to a lower learning rate (1e-5, full model unfrozen or top-only), run for 3–5 epochs, and stop right at the first validation plateau. Treat this as a refresh, not a full re-train from scratch.

Suggested next steps

1. 1.Fine-tune using class_weights.json for 3–5 epochs.
2. 2.Re-evaluate against a held-out slice of our ORIGINAL 134k validation set (never train on this slice!) to confirm the weak classes actually improved and the strong ones didn't regress.
3. 3.Keep snapping field photos for the 52 uncovered classes to fill out our dataset for future iterations.