suryadv/iNat21_filtered

iNat21 manually filtered masks

This repository contains the original iNat21 mask proposals, the manual review decisions, and the filtered mask parquet produced from those decisions.

The masks were reviewed with the ConCor Wildlife Verification Interface, a CPU-only browser interface for stepping through images and mask proposals, accepting or deleting each mask, saving review progress to JSON, and exporting a filtered parquet.

Files

The original proposals came from the iNat21 validation split using the species-balanced source manifest val_species_balanced_subset_2500_seed42.json. The parquet contains masks for 1,669 images and 1,509 species. Source images are not included in this repository.

Manual verification workflow

The review/export flow was:

1. 1.Load original_masks.parquet into the ConCor Wildlife Verification Interface.
2. 2.Review each mask and mark it accepted or deleted. The interface supports the 1 key to accept and advance and the 2 key to delete and advance.
3. 3.Save the complete review state as decisions.json.
4. 4.Export from the interface to create updated_masks.parquet. The interface removes rows marked deleted, keeps rows marked accepted, and does not add decision columns to the parquet.

original_masks.parquet is never modified by this process. Together, the original parquet and decisions.json are the reproducible inputs for generating the filtered parquet again.

Regenerate updated_masks.parquet

Clone and install the verification interface:

git clone https://github.com/suryathecreator/ConCor-Wildlife-Verification-Interface.git
cd ConCor-Wildlife-Verification-Interface
/usr/bin/python3.12 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
mkdir -p data edits exports

Download the two reproducibility inputs from this dataset:

hf download suryadv/iNat21_filtered \
  original_masks.parquet decisions.json \
  --repo-type dataset \
  --local-dir ./inat21_filtered_artifacts

cp ./inat21_filtered_artifacts/original_masks.parquet data/masks.parquet
cp ./inat21_filtered_artifacts/decisions.json edits/decisions.json

Start the verification interface and pass decisions.json back through it with the original parquet:

python server.py \
  --parquet data/masks.parquet \
  --decisions edits/decisions.json \
  --image-root /path/to/SpeciesDataset_iNat21/data \
  --host 127.0.0.1 \
  --port 8000

Open http://127.0.0.1:8000, then select Export + Download Bundle. The interface first saves the loaded decisions and then writes exports/updated_masks.parquet. The image root is needed to display source photographs during review; the stored file_name values are relative paths such as val/<class>/<image>.jpg.

On the original Hyak workspace, the image root was:

/gscratch/krishna/suryadv/SpeciesDataset_iNat21/data

The source_parquet field inside decisions.json records the path used on the review machine. It is provenance metadata; use the downloaded original_masks.parquet as the actual input when reproducing the export.

Integrity