maximilian-franz/basil-instances

maximilian-franz/basil-instances

Per-plant-instance segmented crops derived from ['maximilian-franz/basil', 'maximilian-franz/basil-2'], one row per (original frame × confirmed plant instance).

Layout

ImageFolder-style dataset: metadata.csv at the repo root, with a file_name column pointing to each row's masked crop under images/<plant_instance_id>/<NNNN>.png, and a bbox_file_name column pointing to the same row's plain rectangular crop under images_bbox/<plant_instance_id>/<NNNN>.png. Load with:

from datasets import load_dataset
ds = load_dataset("maximilian-franz/basil-instances")

Columns

• —file_name, relative path to the masked crop (see Layout above)
• —bbox_file_name, relative path to the unmasked rectangular crop of the same padded box
• —trial_id, which of the discovered growing trials this row belongs to (see Known limitations below); the downstream anomaly-detection model's leave-one-trial-out cross-validation folds are one trial each
• —camera_id, tower_id, timestamp, carried over from the source frame
• —plant_instance_id, {trial_id}_{tower_id}_{camera_id}_pos{n}, n assigned by angle-sorting confirmed t0 centroids clockwise from 12 o'clock around their group's centroid
• —bbox_x, bbox_y, bbox_w, bbox_h, instance bounding box in the original (uncropped) frame's pixel coordinates, before the 20px padding applied to the crops
• —mask_iou_score, sigmoid of the SAM 3 tracker's per-frame object-score logit for this instance; a stability/confidence proxy, not IoU against ground truth
• —flagged_for_review, True if this (frame, instance) tripped the merge-detection heuristics below
• —identity_cross_check_disagreement, True if an appearance-embedding cross-check disagreed with a merge correction written for this row (see the identity cross-check bullet below); distinct from flagged_for_review, which is already True on every corrected row regardless of this outcome

Segmentation & tracking

• —Model: SAM 3 (facebook/sam3), via transformers (Sam3Model/Sam3Processor for candidate discovery, Sam3TrackerVideoModel/ Sam3TrackerVideoProcessor for video propagation).
• —Candidate text prompt: "basil seedling" (pcs_score_threshold=0.5, pcs_mask_threshold=0.5); heuristic greenness-threshold + connected-components fallback when unavailable.
• —Manual review: instances confirmed via an in-notebook widget at each of the 8 (tower_id, camera_id) groups' earliest frame; 2346/12495 (18.8%) of output rows were flagged for review by the merge-detection heuristics below.
• —Merge-detection thresholds: an age-indexed growth budget fit per plant instance from its own area trajectory, flagged as a spike above cfg.merge_growth_upper_ratio (1.75×) or a collapse below cfg.merge_growth_lower_ratio (0.4×) the fitted expected area at that frame's age; pairwise mask IoU threshold 0.05. Independently, video propagation (Section 5) clips every propagated mask each frame to its own last-known bounding box expanded by cfg.propagation_clip_margin_px (40px) as a preventative safety valve, so an absorbed neighboring-plant region is stopped from entering the propagated mask in the first place rather than only being caught after the fact by the thresholds above. Flagged growth-budget-spike + IoU-flag corrections are attempted through a three-tier chain, in order: (1) Gemma 4 31B IT's own geometric read of a color-overlay crop of the two flagged masks (attempt_gemma_merge_correction), a separate, hosted-API mechanism, not a SAM 3 one; (2) a SAM 3 detector-assisted reseed from a fresh local detection (detector_assisted_correction), inspired by the detector-assisted re-prompting strategy of "Re-Prompting SAM 3 via Object Retrieval" (arXiv:2603.23788); here, candidate identity is resolved using spatial proximity rather than DINOv3-based appearance retrieval; (3) if both fail, a hand-placed negative point at the neighbor's last-known centroid. Pure area-collapse flags (no neighbor to seed a correction from) get a single-frame detector-assisted patch attempt of their own instead.
• —Identity cross-check: a narrow-scope appearance-embedding signal (MobileNetV3-Small, masked-crop features), consulted only when a growth-budget/IoU flag has already fired, never a continuous per-frame check, comparing a correction's resulting mask against each involved instance's own recent embedding history as a 2x2 assignment problem (identity_cross_check). Disagreement is recorded in identity_cross_check_disagreement (see Columns above) and does not block the correction from being written.
• —Crop format: bounding box padded by 20px, shipped as two crops per row: the mask-zeroed crop (file_name), with pixels outside the instance mask zero-filled to avoid leaking neighboring-plant pixels into the image branch of the downstream anomaly-detection model, and the plain rectangular crop (bbox_file_name), with no masking applied, useful wherever the raw unmasked appearance inside the box matters (e.g. visual QA, or any downstream use that doesn't want the mask's zero-fill artifact).

Known limitations

• —Stereo cross-validation between a tower's two cameras is out of scope for this pipeline, each (tower_id, camera_id) sequence is segmented and propagated fully independently.
• —Degenerate/empty bounding boxes in the source data: 0 row(s).
• —clean_binary_mask (Section 4) unconditionally keeps only one connected component per instance, so a real leaf that is genuinely disconnected from the rest of the plant in a given frame's 2D silhouette (e.g. an occluded connecting stem) would be discarded as if it were a neighbor's stray sliver, an unvalidated assumption not checked against this dataset directly.
• —trial_id is a bare sequential integer (0, 1, 2, ...), assigned in the order trials were discovered and confirmed via TrialBoundaryApp (Section 3), not a calendar date. The actual start/end timestamp for each trialid is not repeated in this dataset; it lives in `trialboundaries.json` from the segmentation pipeline's own repo, if that mapping is needed.